[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:06 dist-test-slave-cnv3 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:06 dist-test-slave-cnv3 krb5kdc[399](info): set up 2 sockets
Sep 24 07:11:06 dist-test-slave-cnv3 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 07:11:07.096001 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:07.220957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:07.332363 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:07.332676 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:07.332734 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:07.337003 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:07.337075 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:07.337103 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:07.337126 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:07.337152 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:07.343240 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:07.344462 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:07.345577 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:07.354833 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:07.354939 412 server_base.cc:1034] running on GCE node
W20260924 07:11:07.354861 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:07.354839 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:07.355741 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:07.357170 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:07.358350 412 hybrid_clock.cc:648] HybridClock initialized: now 1790233867358277 us; error 88 us; skew 500 ppm
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:07.361197 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:07.362155 412 webserver.cc:460] Webserver started at http://127.0.98.62:46213/ using document root <none> and password file <none>
I20260924 07:11:07.362712 412 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:07.362776 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:07.362977 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:07.365041 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "2f28f66211134f6eab6cc54012944af2"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "b8c27acf51e7f801e32d0a6add0d09e0"
server_key_iv: "69b279577dc505bae519eb6d71cafb01"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.365573 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f28f66211134f6eab6cc54012944af2"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "b8c27acf51e7f801e32d0a6add0d09e0"
server_key_iv: "69b279577dc505bae519eb6d71cafb01"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.369329 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 07:11:07.371776 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.373080 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:07.373196 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "2f28f66211134f6eab6cc54012944af2"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "b8c27acf51e7f801e32d0a6add0d09e0"
server_key_iv: "69b279577dc505bae519eb6d71cafb01"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.373291 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:07.410538 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:07.414620 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:07.414904 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:07.423226 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36521
I20260924 07:11:07.423308 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36521 every 8 connection(s)
I20260924 07:11:07.424283 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:07.427345 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:07.428066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 412
I20260924 07:11:07.428190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:07.429466 392 external_mini_cluster.cc:1428] Setting key 92e850e57bcdd22bc9072040f72723ca
I20260924 07:11:07.433992 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2: Bootstrap starting.
I20260924 07:11:07.436430 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:07.437224 480 log.cc:824] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:07.439579 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2: No bootstrap required, opened a new log
I20260924 07:11:07.443437 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260924 07:11:07.443719 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:07.443776 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f28f66211134f6eab6cc54012944af2, State: Initialized, Role: FOLLOWER
I20260924 07:11:07.444219 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260924 07:11:07.444353 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:07.444419 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:07.444531 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:07.445839 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260924 07:11:07.446206 480 leader_election.cc:302] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f28f66211134f6eab6cc54012944af2; no voters:
I20260924 07:11:07.446527 480 leader_election.cc:288] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:07.446651 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:07.446903 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [term 1 LEADER]: Becoming Leader. State: Replica: 2f28f66211134f6eab6cc54012944af2, State: Running, Role: LEADER
I20260924 07:11:07.447268 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } }
I20260924 07:11:07.448026 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:07.448961 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f28f66211134f6eab6cc54012944af2. Latest consensus state: current_term: 1 leader_uuid: "2f28f66211134f6eab6cc54012944af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260924 07:11:07.449095 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:07.449419 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f28f66211134f6eab6cc54012944af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f28f66211134f6eab6cc54012944af2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36521 } } }
I20260924 07:11:07.449506 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:07.449900 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:07.453115 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:07.458300 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:07.432840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.0.98.62:36521)
0924 07:11:07.433328 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:07.433343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:07.434512 (+ 1169us) server_negotiation.cc:407] Connection header received
0924 07:11:07.438088 (+ 3576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:07.438113 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:07.438586 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:07.438976 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:07.442112 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.443157 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.444712 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.445430 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.445621 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:07.451168 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:07.451175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:07.451187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:07.451218 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:07.453236 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.454018 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.454025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.454031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.454175 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.454533 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.454536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.454538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.454945 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:07.455143 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:07.457090 (+ 1947us) server_negotiation.cc:299] Negotiation successful
0924 07:11:07.457403 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":141,"threads_started":1}
I20260924 07:11:07.460037 494 catalog_manager.cc:1347] Generated new cluster ID: 3ea62b4dbb5d456aa00c4e0212cafd4c
I20260924 07:11:07.460104 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:07.475422 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:07.476563 494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:07.482465 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f28f66211134f6eab6cc54012944af2: Generated new TSK 0
I20260924 07:11:07.483436 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:07.519169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:07.635660 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:07.635941 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:07.635999 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:07.640118 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:07.640187 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:07.640254 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:07.645694 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:07.646773 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:07.647850 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:07.657559 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:07.657549 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:07.657642 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:07.657583 506 server_base.cc:1034] running on GCE node
I20260924 07:11:07.658381 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:07.659103 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:07.660284 506 hybrid_clock.cc:648] HybridClock initialized: now 1790233867660243 us; error 52 us; skew 500 ppm
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:07.663187 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:07.664180 506 webserver.cc:460] Webserver started at http://127.0.98.1:35807/ using document root <none> and password file <none>
I20260924 07:11:07.664739 506 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:07.664806 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:07.664995 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:07.667083 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7e8cd7ef6a74844b7f54412b44d7225"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "f416cbce59fd7be0a5cc2eb62a37d925"
server_key_iv: "32e94b8dee670e41e44e12e5db92c380"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.667676 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7e8cd7ef6a74844b7f54412b44d7225"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "f416cbce59fd7be0a5cc2eb62a37d925"
server_key_iv: "32e94b8dee670e41e44e12e5db92c380"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.671505 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:07.674101 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.675479 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:07.675599 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "a7e8cd7ef6a74844b7f54412b44d7225"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "f416cbce59fd7be0a5cc2eb62a37d925"
server_key_iv: "32e94b8dee670e41e44e12e5db92c380"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.675709 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:07.700564 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:07.703938 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:07.704138 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:07.704718 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:07.705585 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:07.705636 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.705689 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:07.705719 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.715212 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37171
I20260924 07:11:07.715242 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37171 every 8 connection(s)
I20260924 07:11:07.716174 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:07.726052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 506
I20260924 07:11:07.726166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:07.726475 392 external_mini_cluster.cc:1428] Setting key de3ce1e473d751ca8fe6049c001df30f
I20260924 07:11:07.730469 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:07.718104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47863 (local address 127.0.98.62:36521)
0924 07:11:07.718240 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:07.718243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:07.719195 (+ 952us) server_negotiation.cc:407] Connection header received
0924 07:11:07.720653 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:07.720656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:07.720713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:07.720783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:07.722003 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.722530 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.723696 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.723971 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.724062 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:07.726712 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:07.726716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:07.726718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:07.726743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:07.728341 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.728870 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.728874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.728875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.728960 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.729280 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.729284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.729285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.729522 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:07.729620 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:07.730169 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 07:11:07.730302 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:11:07.731326 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:07.718528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.1:47863)
0924 07:11:07.719000 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:07.719044 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:07.720403 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:07.720973 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:07.720981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:07.721358 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:07.721853 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:07.721867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.722655 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.722657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:07.723581 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:07.723590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.724122 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.724125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:07.724287 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:07.724854 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:07.724871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:07.726525 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:07.728484 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:07.728491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:07.728506 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:07.728773 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:07.729042 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:07.729045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:07.729048 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:07.729194 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:07.729663 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:07.729670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:07.729926 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 07:11:07.730113 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 07:11:07.730421 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260924 07:11:07.732596 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260924 07:11:07.732878 635 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:07.733510 635 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260924 07:11:07.735241 444 ts_manager.cc:194] Registered new tserver with Master: a7e8cd7ef6a74844b7f54412b44d7225 (127.0.98.1:37171)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:07.736593 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:07.754884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:07.867231 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:07.867537 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:07.867584 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:07.871763 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:07.871835 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:07.871898 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:07.877315 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:07.878433 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:07.879544 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:07.889540 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:07.889544 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:07.889540 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:07.889595 640 server_base.cc:1034] running on GCE node
I20260924 07:11:07.890128 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:07.890755 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:07.891907 640 hybrid_clock.cc:648] HybridClock initialized: now 1790233867891891 us; error 31 us; skew 500 ppm
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:07.894699 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:07.895612 640 webserver.cc:460] Webserver started at http://127.0.98.2:33483/ using document root <none> and password file <none>
I20260924 07:11:07.896162 640 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:07.896227 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:07.896413 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:07.898511 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "4ed182c199a04cf8832a0d099df32fcd"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "2b02ae09be132cc25af97852f5fec283"
server_key_iv: "1096d36cfaf54e673b092d795c6f04bc"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.899094 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4ed182c199a04cf8832a0d099df32fcd"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "2b02ae09be132cc25af97852f5fec283"
server_key_iv: "1096d36cfaf54e673b092d795c6f04bc"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.903093 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:07.905786 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.907107 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:11:07.907230 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "4ed182c199a04cf8832a0d099df32fcd"
format_stamp: "Formatted at 2026-09-24 07:11:07 on dist-test-slave-cnv3"
server_key: "2b02ae09be132cc25af97852f5fec283"
server_key_iv: "1096d36cfaf54e673b092d795c6f04bc"
server_key_version: "encryptionkey@0"
I20260924 07:11:07.907330 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:07.930538 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:07.934325 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:07.934546 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:07.935256 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:07.936338 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:07.936399 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.936462 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:07.936491 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:07.947718 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34323
I20260924 07:11:07.947738 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34323 every 8 connection(s)
I20260924 07:11:07.948753 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:07.953897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 640
I20260924 07:11:07.954015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:07.954272 392 external_mini_cluster.cc:1428] Setting key 01288423943906e870d35278dfd4e8a9
Sep 24 07:11:07 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:07.965656 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:07.951216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57617 (local address 127.0.98.62:36521)
0924 07:11:07.951364 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:07.951368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:07.952464 (+ 1096us) server_negotiation.cc:407] Connection header received
0924 07:11:07.954293 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:07.954297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:07.954355 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:07.954450 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:07.956172 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.956927 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.958464 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.958785 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.958856 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:07.961990 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:07.961993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:07.961995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:07.962095 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:07.963729 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.964318 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.964320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.964322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.964393 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:07.964694 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:07.964696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:07.964697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:07.964865 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:07.964931 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:07.965430 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 07:11:07.965534 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:11:07.966580 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:07.951733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.2:57617)
0924 07:11:07.952295 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:07.952358 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:07.954038 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:07.954663 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:07.954673 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:07.955196 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:07.955981 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:07.956003 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.957092 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.957095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:07.958325 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:07.958336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:07.959078 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:07.959081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:07.959267 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:07.960024 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:07.960048 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:07.961809 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:07.963846 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:07.963853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:07.963870 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:07.964222 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:07.964469 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:07.964472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:07.964473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:07.964618 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:07.964934 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:07.964939 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:07.965216 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 07:11:07.965373 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:11:07.965678 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:07.967933 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260924 07:11:07.968209 769 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:07.968786 769 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260924 07:11:07.969841 444 ts_manager.cc:194] Registered new tserver with Master: 4ed182c199a04cf8832a0d099df32fcd (127.0.98.2:34323)
I20260924 07:11:07.970373 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:07.983735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:08.095971 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:08.096266 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:08.096329 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:08.100443 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:08.100514 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:08.100597 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:08.106192 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36521
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:08.107425 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:08.108466 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:08.118369 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:08.118376 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:08.118369 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:08.118432 774 server_base.cc:1034] running on GCE node
I20260924 07:11:08.119073 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:08.119764 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:08.120926 774 hybrid_clock.cc:648] HybridClock initialized: now 1790233868120896 us; error 42 us; skew 500 ppm
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:08.124568 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:08.125675 774 webserver.cc:460] Webserver started at http://127.0.98.3:39445/ using document root <none> and password file <none>
I20260924 07:11:08.126291 774 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:08.126368 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:08.126559 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:08.128769 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "236fe15dd2d946d0903a8e90a7a2dee7"
format_stamp: "Formatted at 2026-09-24 07:11:08 on dist-test-slave-cnv3"
server_key: "71744f2ff08ab0a285d9071d5b9130fd"
server_key_iv: "cbb9242011d2c73ee2335758c232cdd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:08.129374 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "236fe15dd2d946d0903a8e90a7a2dee7"
format_stamp: "Formatted at 2026-09-24 07:11:08 on dist-test-slave-cnv3"
server_key: "71744f2ff08ab0a285d9071d5b9130fd"
server_key_iv: "cbb9242011d2c73ee2335758c232cdd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:08.133452 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:08.136126 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:08.137632 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 07:11:08.137749 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "236fe15dd2d946d0903a8e90a7a2dee7"
format_stamp: "Formatted at 2026-09-24 07:11:08 on dist-test-slave-cnv3"
server_key: "71744f2ff08ab0a285d9071d5b9130fd"
server_key_iv: "cbb9242011d2c73ee2335758c232cdd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:08.137883 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:08.149573 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:08.152949 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:08.153105 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:08.153713 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:08.154565 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:08.154614 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:08.154665 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:08.154695 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:08.164338 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39817
I20260924 07:11:08.164367 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39817 every 8 connection(s)
I20260924 07:11:08.165263 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:08.169770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 774
I20260924 07:11:08.169866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:08.170089 392 external_mini_cluster.cc:1428] Setting key 5b5e6505daa09a88aff32d3771bb1ad7
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:08.179589 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:08.167034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48975 (local address 127.0.98.62:36521)
0924 07:11:08.167155 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:08.167158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:08.168118 (+ 960us) server_negotiation.cc:407] Connection header received
0924 07:11:08.169435 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:08.169438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:08.169487 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:08.169575 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:08.170831 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.171547 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.172938 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.173150 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.173227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:08.175556 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:08.175577 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:08.175579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:08.175605 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:08.177416 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.177957 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.177960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.177961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.178040 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.178377 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.178380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.178381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.178594 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:08.178680 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:08.179269 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 07:11:08.179424 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:11:08.180469 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:08.167454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36521 (local address 127.0.98.3:48975)
0924 07:11:08.167946 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:08.168000 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:08.169238 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:08.169732 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:08.169739 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:08.170220 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:08.170710 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:08.170724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.171669 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.171672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:08.172834 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:08.172844 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.173279 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.173282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:08.173418 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:08.173965 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:08.173981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:08.175361 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:08.177576 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:08.177583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:08.177601 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:08.177861 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:08.178128 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:08.178131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:08.178132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:08.178270 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:08.178695 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:08.178700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:08.178951 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 07:11:08.179192 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 07:11:08.179531 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260924 07:11:08.181782 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36521
I20260924 07:11:08.182080 903 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:08.182582 903 heartbeater.cc:511] Master 127.0.98.62:36521 requested a full tablet report, sending...
I20260924 07:11:08.183717 444 ts_manager.cc:194] Registered new tserver with Master: 236fe15dd2d946d0903a8e90a7a2dee7 (127.0.98.3:39817)
I20260924 07:11:08.184404 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48975
I20260924 07:11:08.184818 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:08.212843 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:08.201371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.98.1:37171)
0924 07:11:08.201776 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:08.201798 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:08.201834 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:08.201900 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:08.201907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:08.202169 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:08.202592 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:08.203354 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.204341 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.205264 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.205733 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.205824 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:08.208630 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:08.208642 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:08.208659 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:08.208693 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:08.210715 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.211183 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.211190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.211191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.211286 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.211656 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.211660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.211661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.211994 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:08.212202 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:08.212448 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 07:11:08.212616 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
W20260924 07:11:08.214025 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44818
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:08.243333 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:08.233194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.0.98.1:37171)
0924 07:11:08.233321 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:08.233325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:08.233455 (+ 130us) server_negotiation.cc:407] Connection header received
0924 07:11:08.233512 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:08.233515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:08.233560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:08.233649 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:08.234480 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.235353 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.236332 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:08.236581 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:08.236669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:08.239296 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:08.239299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:08.239301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:08.239330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:08.240768 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.241315 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.241318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.241319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.241404 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:08.241857 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:08.241861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:08.241863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:08.242296 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:08.242439 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:08.243072 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 07:11:08.243203 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:11:08.244696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 506
I20260924 07:11:08.250547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 640
I20260924 07:11:08.257457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 774
I20260924 07:11:08.262404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 412
2026-09-24T07:11:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2141 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[917](info): set up 2 sockets
Sep 24 07:11:08 dist-test-slave-cnv3 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 07:11:09.222913 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:09 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:09.340941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:09.454468 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:09.454748 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:09.454787 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:09.458782 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:09.458848 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:09.458866 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:09.458881 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:09.458895 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:09.464385 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:09.465425 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:09.466423 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:09.474687 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:09.474707 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:09.474761 930 server_base.cc:1034] running on GCE node
W20260924 07:11:09.474709 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:09.475471 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:09.476516 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:09.477692 930 hybrid_clock.cc:648] HybridClock initialized: now 1790233869477670 us; error 37 us; skew 500 ppm
Sep 24 07:11:09 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:09.480291 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:09.481158 930 webserver.cc:460] Webserver started at http://127.0.98.62:33027/ using document root <none> and password file <none>
I20260924 07:11:09.481688 930 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:09.481748 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:09.481926 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:09.483938 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "78635b1afa96417688a6182b9b0f110e"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "910e0cc1e637cddce4fa1bff44a05d31"
server_key_iv: "3339ac5e036be73b724e6164f84aaf0e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.484457 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "78635b1afa96417688a6182b9b0f110e"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "910e0cc1e637cddce4fa1bff44a05d31"
server_key_iv: "3339ac5e036be73b724e6164f84aaf0e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.487998 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:09.490283 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:09.491556 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:09.491683 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "78635b1afa96417688a6182b9b0f110e"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "910e0cc1e637cddce4fa1bff44a05d31"
server_key_iv: "3339ac5e036be73b724e6164f84aaf0e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.491778 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:09.514325 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:09.517302 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:09.517481 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:09.524410 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39601
I20260924 07:11:09.524447 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39601 every 8 connection(s)
I20260924 07:11:09.525364 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:09.526999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 930
I20260924 07:11:09.527103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:09.527417 392 external_mini_cluster.cc:1428] Setting key bb2426ebcc1de7f6ced031d56e8a771b
I20260924 07:11:09.528568 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:09.534345 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e: Bootstrap starting.
Sep 24 07:11:09 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:09.536887 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:09.537737 998 log.cc:824] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:09.540128 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e: No bootstrap required, opened a new log
I20260924 07:11:09.542497 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:09.529236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.0.98.62:39601)
0924 07:11:09.529706 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:09.529721 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:09.529767 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:11:09.530437 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:09.530454 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:09.530818 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:09.531135 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:09.531935 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.532909 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.533743 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.534231 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.534384 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:09.537293 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:09.537301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:09.537319 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:09.537353 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:09.539341 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:09.539780 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:09.539785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:09.539791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:09.539929 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:09.540200 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:09.540203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:09.540205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:09.540718 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:09.540903 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:09.541185 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 07:11:09.541484 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260924 07:11:09.544462 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260924 07:11:09.544726 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:09.544802 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78635b1afa96417688a6182b9b0f110e, State: Initialized, Role: FOLLOWER
I20260924 07:11:09.545228 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260924 07:11:09.545357 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:09.545404 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:09.545475 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:09.546388 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260924 07:11:09.546763 998 leader_election.cc:302] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78635b1afa96417688a6182b9b0f110e; no voters:
I20260924 07:11:09.547114 998 leader_election.cc:288] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:09.547224 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:09.547474 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [term 1 LEADER]: Becoming Leader. State: Replica: 78635b1afa96417688a6182b9b0f110e, State: Running, Role: LEADER
I20260924 07:11:09.547828 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } }
I20260924 07:11:09.548390 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:09.549541 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78635b1afa96417688a6182b9b0f110e. Latest consensus state: current_term: 1 leader_uuid: "78635b1afa96417688a6182b9b0f110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260924 07:11:09.549541 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78635b1afa96417688a6182b9b0f110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78635b1afa96417688a6182b9b0f110e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39601 } } }
I20260924 07:11:09.549695 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:09.549696 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:09.550076 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:09.553256 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:09.558600 1012 catalog_manager.cc:1347] Generated new cluster ID: 6ec86bf748d94f13ae23f15d40646502
I20260924 07:11:09.558676 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:09.571972 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:09.573385 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:09.583506 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78635b1afa96417688a6182b9b0f110e: Generated new TSK 0
I20260924 07:11:09.584262 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:09.622979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:09.753607 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:09.753892 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:09.753937 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:09.758134 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:09.758213 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:09.758283 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:09.763913 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:09.765053 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:09.766201 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:09.776834 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:09.776827 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:09.776881 1024 server_base.cc:1034] running on GCE node
W20260924 07:11:09.776850 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:09.777486 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:09.778152 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:09.779364 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790233869779333 us; error 45 us; skew 500 ppm
Sep 24 07:11:09 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:09.782230 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:09.783211 1024 webserver.cc:460] Webserver started at http://127.0.98.1:34387/ using document root <none> and password file <none>
I20260924 07:11:09.783855 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:09.783931 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:09.784132 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:09.786237 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c8c778ccc9b45b8aefb7d66749f016d"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "1a4f21f2d9becf4503745895a482e1ac"
server_key_iv: "b7778e0e7589d6e55760dfeab57f1b7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.786810 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8c778ccc9b45b8aefb7d66749f016d"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "1a4f21f2d9becf4503745895a482e1ac"
server_key_iv: "b7778e0e7589d6e55760dfeab57f1b7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.790871 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:09.793535 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:09.794919 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:11:09.795045 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "7c8c778ccc9b45b8aefb7d66749f016d"
format_stamp: "Formatted at 2026-09-24 07:11:09 on dist-test-slave-cnv3"
server_key: "1a4f21f2d9becf4503745895a482e1ac"
server_key_iv: "b7778e0e7589d6e55760dfeab57f1b7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:09.795164 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:09.810035 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:09.813369 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:09.813553 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:09.814175 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:09.815054 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:09.815102 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:09.815158 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:09.815187 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:09.824060 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40031
I20260924 07:11:09.824092 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40031 every 8 connection(s)
I20260924 07:11:09.825086 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:09.830175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1024
I20260924 07:11:09.830296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:09.830534 392 external_mini_cluster.cc:1428] Setting key 30650bd8f394e56f295e72bf8ea8cb86
Sep 24 07:11:09 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:09.841305 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:09.826914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37683 (local address 127.0.98.62:39601)
0924 07:11:09.827065 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:09.827068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:09.828012 (+ 944us) server_negotiation.cc:407] Connection header received
0924 07:11:09.829643 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:09.829647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:09.829710 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:09.829811 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:09.831199 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.831918 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.833145 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.833380 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.833470 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:09.836365 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:09.836369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:09.836371 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:09.836398 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:09.837817 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:09.838549 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:09.838552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:09.838554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:09.838650 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:09.839110 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:09.839113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:09.839115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:09.839499 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:09.839593 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:09.840869 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0924 07:11:09.841047 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:11:09.842371 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:09.827320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.1:37683)
0924 07:11:09.827848 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:09.827897 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:09.829387 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:09.830016 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:09.830024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:09.830483 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:09.831045 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:09.831061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.832066 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.832069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:09.833009 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:09.833019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:09.833531 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:09.833535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:09.833711 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:09.834283 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:09.834300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:09.836089 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:09.837962 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:09.837971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:09.837993 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:09.838399 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:09.838754 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:09.838757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:09.838759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:09.838978 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:09.839624 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:09.839638 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:09.840452 (+ 814us) client_negotiation.cc:769] Sending connection context
0924 07:11:09.840781 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 07:11:09.841118 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":107,"threads_started":1}
I20260924 07:11:09.844146 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260924 07:11:09.844451 1153 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:09.845024 1153 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260924 07:11:09.847026 962 ts_manager.cc:194] Registered new tserver with Master: 7c8c778ccc9b45b8aefb7d66749f016d (127.0.98.1:40031)
I20260924 07:11:09.848368 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260924 07:11:10.660983 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.829s user 0.002s sys 0.001s
I20260924 07:11:10.661523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:10.779966 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:10.780270 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:10.780335 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:10.784616 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:10.784703 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:10.784793 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:10.790653 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:10.792176 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:10.793390 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:10.804817 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:10.804819 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:10.804819 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:10.805274 1158 server_base.cc:1034] running on GCE node
I20260924 07:11:10.805732 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:10.806443 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:10.807612 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790233870807588 us; error 37 us; skew 500 ppm
Sep 24 07:11:10 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:10.810649 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:10.811754 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43737/ using document root <none> and password file <none>
I20260924 07:11:10.812381 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:10.812451 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:10.812650 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:10.814935 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "af68b664f2c94025a766f79c9eb4a738"
format_stamp: "Formatted at 2026-09-24 07:11:10 on dist-test-slave-cnv3"
server_key: "85a0f05817546be1265131fea1721f8a"
server_key_iv: "2b8ed29785a308fac2b54d9cc3dc1ac0"
server_key_version: "encryptionkey@0"
I20260924 07:11:10.815567 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "af68b664f2c94025a766f79c9eb4a738"
format_stamp: "Formatted at 2026-09-24 07:11:10 on dist-test-slave-cnv3"
server_key: "85a0f05817546be1265131fea1721f8a"
server_key_iv: "2b8ed29785a308fac2b54d9cc3dc1ac0"
server_key_version: "encryptionkey@0"
I20260924 07:11:10.819907 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:11:10.822777 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:10.824362 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:10.824499 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "af68b664f2c94025a766f79c9eb4a738"
format_stamp: "Formatted at 2026-09-24 07:11:10 on dist-test-slave-cnv3"
server_key: "85a0f05817546be1265131fea1721f8a"
server_key_iv: "2b8ed29785a308fac2b54d9cc3dc1ac0"
server_key_version: "encryptionkey@0"
I20260924 07:11:10.824611 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:10.838110 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:10.841666 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:10.841861 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:10.842499 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:10.843410 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:10.843472 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:10.843528 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:10.843554 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:10.851644 1153 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260924 07:11:10.853081 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42809
I20260924 07:11:10.853245 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42809 every 8 connection(s)
I20260924 07:11:10.854120 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:10.858228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1158
I20260924 07:11:10.858439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:10.858733 392 external_mini_cluster.cc:1428] Setting key af8ada723d7e41cb0c7b1bd48b5835a0
Sep 24 07:11:10 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:10.868767 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:10.856069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50009 (local address 127.0.98.62:39601)
0924 07:11:10.856348 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:10.856353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:10.857159 (+ 806us) server_negotiation.cc:407] Connection header received
0924 07:11:10.858503 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:10.858507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:10.858568 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:10.858675 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:10.860043 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:10.860677 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:10.862177 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:10.862577 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:10.862681 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:10.865250 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:10.865253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:10.865255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:10.865282 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:10.866673 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:10.867180 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:10.867183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:10.867184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:10.867264 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:10.867649 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:10.867653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:10.867655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:10.867879 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:10.867966 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:10.868447 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 07:11:10.868621 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260924 07:11:10.869592 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:10.856537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.2:50009)
0924 07:11:10.856990 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:10.857059 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:10.858271 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:10.858866 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:10.858875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:10.859314 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:10.859904 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:10.859922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:10.860811 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:10.860815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:10.862066 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:10.862075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:10.862717 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:10.862719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:10.862863 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:10.863524 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:10.863545 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:10.865072 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:10.866789 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:10.866795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:10.866814 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:10.867083 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:10.867347 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:10.867350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:10.867352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:10.867545 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:10.867969 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:10.867975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:10.868240 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 07:11:10.868402 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 07:11:10.868713 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260924 07:11:10.870772 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260924 07:11:10.871011 1287 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:10.871542 1287 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260924 07:11:10.872737 962 ts_manager.cc:194] Registered new tserver with Master: af68b664f2c94025a766f79c9eb4a738 (127.0.98.2:42809)
I20260924 07:11:10.873322 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:11.875505 1287 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 07:11:11.879900 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.019s user 0.000s sys 0.003s
I20260924 07:11:11.880360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:11.992240 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:11.992522 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:11.992583 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:11.996675 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:11.996735 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:11.996814 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:12.002246 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39601
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:12.003347 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:12.004423 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:12.015146 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:12.015169 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:12.015270 1293 server_base.cc:1034] running on GCE node
W20260924 07:11:12.015208 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:12.015949 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:12.016640 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:12.017851 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790233872017811 us; error 55 us; skew 500 ppm
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:12.025488 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:12.026551 1293 webserver.cc:460] Webserver started at http://127.0.98.3:43017/ using document root <none> and password file <none>
I20260924 07:11:12.027108 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:12.027168 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:12.027334 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:12.029608 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "38c149183a4e4e1cbbcee89b7e6a36a6"
format_stamp: "Formatted at 2026-09-24 07:11:12 on dist-test-slave-cnv3"
server_key: "dbfb5b0456b79b67dc4d1fe9b078ca87"
server_key_iv: "5c4ce656abbe3ae4a48963a96a7cf627"
server_key_version: "encryptionkey@0"
I20260924 07:11:12.030169 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38c149183a4e4e1cbbcee89b7e6a36a6"
format_stamp: "Formatted at 2026-09-24 07:11:12 on dist-test-slave-cnv3"
server_key: "dbfb5b0456b79b67dc4d1fe9b078ca87"
server_key_iv: "5c4ce656abbe3ae4a48963a96a7cf627"
server_key_version: "encryptionkey@0"
I20260924 07:11:12.034111 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:12.036578 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:12.037808 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:12.037904 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "38c149183a4e4e1cbbcee89b7e6a36a6"
format_stamp: "Formatted at 2026-09-24 07:11:12 on dist-test-slave-cnv3"
server_key: "dbfb5b0456b79b67dc4d1fe9b078ca87"
server_key_iv: "5c4ce656abbe3ae4a48963a96a7cf627"
server_key_version: "encryptionkey@0"
I20260924 07:11:12.037997 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:12.062716 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:12.065807 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:12.065968 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:12.066608 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:12.067474 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:12.067529 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:12.067575 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:12.067600 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:12.077327 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39289
I20260924 07:11:12.077351 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39289 every 8 connection(s)
I20260924 07:11:12.078266 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:12.086638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1293
I20260924 07:11:12.086772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:12.087067 392 external_mini_cluster.cc:1428] Setting key f1d1712e7c9db14df66735c39a52e0ad
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:12.092886 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.080290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57465 (local address 127.0.98.62:39601)
0924 07:11:12.080530 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.080534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.081398 (+ 864us) server_negotiation.cc:407] Connection header received
0924 07:11:12.083003 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.083006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.083059 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.083145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:12.084466 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.085076 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.086315 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.086548 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.086637 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.089558 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:12.089561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:12.089563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:12.089593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:12.091018 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.091520 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.091523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.091524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.091603 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.091878 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.091880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.091881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.092075 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:12.092146 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.092623 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.092752 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260924 07:11:12.093750 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.080800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39601 (local address 127.0.98.3:57465)
0924 07:11:12.081248 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.081293 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.082770 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.083341 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.083349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.083789 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:12.084302 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.084317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.085181 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.085183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.086189 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.086199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.086716 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.086720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.086920 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.087537 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:12.087558 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:12.089410 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:12.091130 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:12.091134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:12.091147 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:12.091439 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:12.091674 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:12.091676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:12.091677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:12.091807 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:12.092146 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:12.092151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:12.092418 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.092570 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.092846 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260924 07:11:12.094863 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39601
I20260924 07:11:12.095108 1422 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:12.095669 1422 heartbeater.cc:511] Master 127.0.98.62:39601 requested a full tablet report, sending...
I20260924 07:11:12.096786 962 ts_manager.cc:194] Registered new tserver with Master: 38c149183a4e4e1cbbcee89b7e6a36a6 (127.0.98.3:39289)
I20260924 07:11:12.097352 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57465
I20260924 07:11:12.101014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:11:12.111963 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.104606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.0.98.62:39601)
0924 07:11:12.104807 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.104812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.104937 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:11:12.105045 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.105047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.105091 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.105161 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:12.105851 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.106405 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.107258 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.107505 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.107589 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.108910 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:12.108913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:12.108915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:12.108944 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:12.110368 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.110762 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.110764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.110765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.110850 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.111046 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.111048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.111049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.111282 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:12.111413 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.111691 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.111797 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 07:11:12.117359 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:12.120576 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:12.136122 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.129976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39289 (local address 127.0.0.1:36640)
0924 07:11:12.130788 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.130842 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.131104 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.131842 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.131848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.131870 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:12.132127 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.132137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133255 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.133258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.134435 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.134446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.135563 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.135566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.135678 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.135715 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.135826 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.135962 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"spinlock_wait_cycles":10880,"thread_start_us":134,"threads_started":1}
I20260924 07:11:12.136129 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.130107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.0.98.3:39289)
0924 07:11:12.130473 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.130484 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.130939 (+ 455us) server_negotiation.cc:407] Connection header received
0924 07:11:12.131300 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.131306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.131557 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.131721 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:12.132262 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133139 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.134574 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.135469 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.135566 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.135731 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.135867 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.135962 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":145,"threads_started":1}
I20260924 07:11:12.136822 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.130712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42809 (local address 127.0.0.1:50836)
0924 07:11:12.131603 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.131618 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.131706 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.132318 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.132322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.132338 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:12.132436 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.132442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133860 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.133863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.134900 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.134909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.136071 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.136074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.136155 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.136180 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.136591 (+ 411us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.136679 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":24,"thread_start_us":94,"threads_started":1}
I20260924 07:11:12.136843 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.130043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40031 (local address 127.0.0.1:38894)
0924 07:11:12.130927 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.130942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.131067 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.132394 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.132397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.132415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:12.132495 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.132501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133843 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.133847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.135204 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.135214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.136465 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.136469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.136570 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.136597 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.136682 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.136744 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":46,"spinlock_wait_cycles":2560,"thread_start_us":84,"threads_started":1}
I20260924 07:11:12.136991 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.131113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.98.1:40031)
0924 07:11:12.132009 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.132015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.132035 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:11:12.132100 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.132105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.132275 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.132410 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:12.132587 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133728 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.135332 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.136144 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.136290 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.136422 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.136736 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.136858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":86,"threads_started":1}
I20260924 07:11:12.137521 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.130876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50836 (local address 127.0.98.2:42809)
0924 07:11:12.131249 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.131256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.131792 (+ 536us) server_negotiation.cc:407] Connection header received
0924 07:11:12.131867 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.131872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.132056 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.132189 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:12.132553 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.133731 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.135023 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.135780 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.135879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.136011 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.137329 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.137418 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":91,"threads_started":1}
I20260924 07:11:12.139940 1356 tablet_service.cc:1467] Processing CreateTablet for tablet dd48bdfe98534e22b2c0f73218027561 (DEFAULT_TABLE table=test-table [id=ea77a5abdaf3455b90b558a428431c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:12.141206 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48bdfe98534e22b2c0f73218027561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:12.141669 1087 tablet_service.cc:1467] Processing CreateTablet for tablet dd48bdfe98534e22b2c0f73218027561 (DEFAULT_TABLE table=test-table [id=ea77a5abdaf3455b90b558a428431c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:12.142764 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48bdfe98534e22b2c0f73218027561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:12.145761 1221 tablet_service.cc:1467] Processing CreateTablet for tablet dd48bdfe98534e22b2c0f73218027561 (DEFAULT_TABLE table=test-table [id=ea77a5abdaf3455b90b558a428431c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:12.146733 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48bdfe98534e22b2c0f73218027561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:12.149752 1441 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Bootstrap starting.
I20260924 07:11:12.151087 1442 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Bootstrap starting.
I20260924 07:11:12.152290 1441 tablet_bootstrap.cc:654] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:12.153395 1442 tablet_bootstrap.cc:654] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:12.153560 1443 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Bootstrap starting.
I20260924 07:11:12.153637 1441 log.cc:824] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:12.154196 1442 log.cc:824] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:12.155689 1441 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: No bootstrap required, opened a new log
I20260924 07:11:12.155875 1441 ts_tablet_manager.cc:1397] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:11:12.156260 1442 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: No bootstrap required, opened a new log
I20260924 07:11:12.156272 1443 tablet_bootstrap.cc:654] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:12.156451 1442 ts_tablet_manager.cc:1397] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:12.157064 1443 log.cc:824] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:12.159487 1443 tablet_bootstrap.cc:492] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: No bootstrap required, opened a new log
I20260924 07:11:12.159711 1443 ts_tablet_manager.cc:1397] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:11:12.161284 1442 raft_consensus.cc:348] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.161593 1442 raft_consensus.cc:374] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:12.161659 1442 raft_consensus.cc:729] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c8c778ccc9b45b8aefb7d66749f016d, State: Initialized, Role: FOLLOWER
I20260924 07:11:12.161679 1441 raft_consensus.cc:348] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.161929 1441 raft_consensus.cc:374] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:12.161981 1441 raft_consensus.cc:729] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c149183a4e4e1cbbcee89b7e6a36a6, State: Initialized, Role: FOLLOWER
I20260924 07:11:12.162130 1442 consensus_queue.cc:260] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.162384 1441 consensus_queue.cc:260] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.163264 1442 ts_tablet_manager.cc:1428] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 07:11:12.163348 1441 ts_tablet_manager.cc:1428] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:11:12.163695 1422 heartbeater.cc:503] Master 127.0.98.62:39601 was elected leader, sending a full tablet report...
I20260924 07:11:12.164798 1443 raft_consensus.cc:348] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.165037 1443 raft_consensus.cc:374] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:12.165093 1443 raft_consensus.cc:729] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af68b664f2c94025a766f79c9eb4a738, State: Initialized, Role: FOLLOWER
I20260924 07:11:12.165500 1443 consensus_queue.cc:260] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.166467 1443 ts_tablet_manager.cc:1428] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:11:12.317032 1449 raft_consensus.cc:482] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:12.317266 1449 raft_consensus.cc:504] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.318601 1449 leader_election.cc:288] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38c149183a4e4e1cbbcee89b7e6a36a6 (127.0.98.3:39289), 7c8c778ccc9b45b8aefb7d66749f016d (127.0.98.1:40031)
I20260924 07:11:12.322485 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.318956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56461 (local address 127.0.98.1:40031)
0924 07:11:12.319073 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.319076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.319526 (+ 450us) server_negotiation.cc:407] Connection header received
0924 07:11:12.319587 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.319590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.319639 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.319699 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:12.319956 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.320465 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.321567 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.321984 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.322070 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.322131 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.322291 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.322355 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:11:12.322541 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.319098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44807 (local address 127.0.98.3:39289)
0924 07:11:12.319239 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.319243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.319359 (+ 116us) server_negotiation.cc:407] Connection header received
0924 07:11:12.319525 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.319528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.319569 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.319631 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:12.319927 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.320451 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.321637 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.322038 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.322128 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.322187 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.322353 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.322427 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:11:12.322523 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.318862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40031 (local address 127.0.98.2:56461)
0924 07:11:12.319362 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.319396 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.319509 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.319720 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.319723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.319737 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:12.319859 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.319864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.320560 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.320564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.321446 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.321455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.322084 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.322088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.322180 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.322202 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.322267 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.322333 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":238,"thread_start_us":119,"threads_started":1}
I20260924 07:11:12.322566 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.318862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39289 (local address 127.0.98.2:44807)
0924 07:11:12.319268 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:12.319280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:12.319418 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:12.319657 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:12.319659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:12.319674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:12.319827 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.319833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.320556 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.320559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.321519 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:12.321528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.322147 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.322151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:12.322252 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.322268 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:11:12.322331 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:11:12.322403 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":66,"thread_start_us":127,"threads_started":1}
I20260924 07:11:12.323447 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48bdfe98534e22b2c0f73218027561" candidate_uuid: "af68b664f2c94025a766f79c9eb4a738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" is_pre_election: true
I20260924 07:11:12.323438 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48bdfe98534e22b2c0f73218027561" candidate_uuid: "af68b664f2c94025a766f79c9eb4a738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" is_pre_election: true
I20260924 07:11:12.323729 1107 raft_consensus.cc:2454] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af68b664f2c94025a766f79c9eb4a738 in term 0.
I20260924 07:11:12.323735 1376 raft_consensus.cc:2454] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af68b664f2c94025a766f79c9eb4a738 in term 0.
I20260924 07:11:12.324186 1174 leader_election.cc:302] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38c149183a4e4e1cbbcee89b7e6a36a6, af68b664f2c94025a766f79c9eb4a738; no voters:
I20260924 07:11:12.324441 1449 raft_consensus.cc:2781] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:12.324534 1449 raft_consensus.cc:482] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:12.324591 1449 raft_consensus.cc:3037] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 0 FOLLOWER]: Advancing to term 1
W20260924 07:11:12.327759 1154 tablet.cc:2369] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:12.329224 1423 tablet.cc:2369] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:12.329200 1449 raft_consensus.cc:504] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.329674 1449 leader_election.cc:288] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [CANDIDATE]: Term 1 election: Requested vote from peers 38c149183a4e4e1cbbcee89b7e6a36a6 (127.0.98.3:39289), 7c8c778ccc9b45b8aefb7d66749f016d (127.0.98.1:40031)
I20260924 07:11:12.330180 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48bdfe98534e22b2c0f73218027561" candidate_uuid: "af68b664f2c94025a766f79c9eb4a738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c8c778ccc9b45b8aefb7d66749f016d"
I20260924 07:11:12.330328 1107 raft_consensus.cc:3037] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:12.330369 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48bdfe98534e22b2c0f73218027561" candidate_uuid: "af68b664f2c94025a766f79c9eb4a738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6"
I20260924 07:11:12.330503 1376 raft_consensus.cc:3037] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:12.331482 1107 raft_consensus.cc:2454] T dd48bdfe98534e22b2c0f73218027561 P 7c8c778ccc9b45b8aefb7d66749f016d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af68b664f2c94025a766f79c9eb4a738 in term 1.
I20260924 07:11:12.331750 1376 raft_consensus.cc:2454] T dd48bdfe98534e22b2c0f73218027561 P 38c149183a4e4e1cbbcee89b7e6a36a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af68b664f2c94025a766f79c9eb4a738 in term 1.
I20260924 07:11:12.331820 1175 leader_election.cc:302] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c8c778ccc9b45b8aefb7d66749f016d, af68b664f2c94025a766f79c9eb4a738; no voters:
I20260924 07:11:12.332178 1449 raft_consensus.cc:2781] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:12.332427 1449 raft_consensus.cc:686] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [term 1 LEADER]: Becoming Leader. State: Replica: af68b664f2c94025a766f79c9eb4a738, State: Running, Role: LEADER
I20260924 07:11:12.332808 1449 consensus_queue.cc:237] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } }
I20260924 07:11:12.336422 960 catalog_manager.cc:5495] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738 reported cstate change: term changed from 0 to 1, leader changed from <none> to af68b664f2c94025a766f79c9eb4a738 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "af68b664f2c94025a766f79c9eb4a738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38c149183a4e4e1cbbcee89b7e6a36a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c8c778ccc9b45b8aefb7d66749f016d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af68b664f2c94025a766f79c9eb4a738" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42809 } health_report { overall_health: HEALTHY } } }
W20260924 07:11:12.356098 1288 tablet.cc:2369] T dd48bdfe98534e22b2c0f73218027561 P af68b664f2c94025a766f79c9eb4a738: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:12.374284 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.365836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.98.1:40031)
0924 07:11:12.365964 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.365969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.366126 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:11:12.366258 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.366260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.366303 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.366368 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:12.367091 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.367635 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.368463 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.368681 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.368777 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.371140 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:12.371154 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:12.371169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:12.371194 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:12.372837 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.373237 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.373243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.373244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.373322 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.373516 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.373518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.373519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.373762 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:12.373913 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.374045 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.374143 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:12.421200 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.412860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.1:40031)
0924 07:11:12.412998 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.413003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.413179 (+ 176us) server_negotiation.cc:407] Connection header received
0924 07:11:12.413317 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.413320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.413376 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.413474 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:12.414199 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.414987 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.415894 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.416116 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.416190 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.418504 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:12.418508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:12.418510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:12.418535 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:12.419772 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.420113 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.420115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.420116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.420178 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.420467 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.420469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.420470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.420625 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:12.420714 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.420980 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.421075 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:11:12 dist-test-slave-cnv3 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:12.495723 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:12.486591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.98.1:40031)
0924 07:11:12.486746 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:12.486751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:12.487032 (+ 281us) server_negotiation.cc:407] Connection header received
0924 07:11:12.487172 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:12.487175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:12.487222 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:12.487301 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:12.488163 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.488681 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.489803 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:12.490017 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:12.490094 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:12.492772 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:12.492775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:12.492777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:12.492803 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:12.494131 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.494501 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.494505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.494506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.494588 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:12.494820 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:12.494822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:12.494823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:12.494982 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:12.495074 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:12.495299 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:11:12.495426 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 07:11:12.497118 1067 tablet_service.cc:2908] Rejecting scan request for tablet dd48bdfe98534e22b2c0f73218027561: Uninitialized: clean time has not yet been initialized
I20260924 07:11:12.499122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1024
I20260924 07:11:12.504477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1158
I20260924 07:11:12.509387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1293
I20260924 07:11:12.514178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 930
2026-09-24T07:11:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4250 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:13 dist-test-slave-cnv3 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:13 dist-test-slave-cnv3 krb5kdc[1466](info): set up 2 sockets
Sep 24 07:11:13 dist-test-slave-cnv3 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260924 07:11:14.701030 392 mini_kdc.cc:121] Time spent starting KDC: real 2.166s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 07:11:14.993855 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.293s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 07:11:15.161919 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.168s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 07:11:15.341171 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.179s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:15 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:15.402655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:15.513013 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:15.513314 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:15.513368 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:15.517439 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:15.517513 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:11:15.517534 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:15.517556 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:15.517577 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:15.517606 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:15.523069 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41789
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:15.524188 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:15.525218 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:11:15.533913 1479 server_base.cc:1034] running on GCE node
W20260924 07:11:15.533885 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:15.533885 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:15.533973 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:15.534740 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:15.535959 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:15.537159 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790233875537131 us; error 44 us; skew 500 ppm
Sep 24 07:11:15 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:15.540259 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:15.541196 1479 webserver.cc:460] Webserver started at http://127.0.98.62:37073/ using document root <none> and password file <none>
I20260924 07:11:15.541734 1479 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:15.541797 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:15.542018 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:15.544234 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "0f78064a58c84d87a7a6db20d7392b3a"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "4d7c25e6db33a2dd7dd6a8cb9a58208a"
server_key_iv: "bacde39c5c59fbbaf6200d44d8d6402c"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.544811 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f78064a58c84d87a7a6db20d7392b3a"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "4d7c25e6db33a2dd7dd6a8cb9a58208a"
server_key_iv: "bacde39c5c59fbbaf6200d44d8d6402c"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.548727 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:15.551301 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.552841 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:15.553002 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "0f78064a58c84d87a7a6db20d7392b3a"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "4d7c25e6db33a2dd7dd6a8cb9a58208a"
server_key_iv: "bacde39c5c59fbbaf6200d44d8d6402c"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.553112 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:15.562373 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:15.565404 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:15.565588 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:15.573410 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41789
I20260924 07:11:15.573457 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41789 every 8 connection(s)
I20260924 07:11:15.574381 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:15.577464 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:15.579226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1479
I20260924 07:11:15.579312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:15.579595 392 external_mini_cluster.cc:1428] Setting key 67560fccf11988f757fc82e1b0720aa0
I20260924 07:11:15.584043 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Bootstrap starting.
I20260924 07:11:15.586619 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:15.587543 1557 log.cc:824] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:11:15 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:15.589663 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: No bootstrap required, opened a new log
I20260924 07:11:15.593371 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260924 07:11:15.593577 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.580970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.98.62:41789)
0924 07:11:15.581356 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.581371 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.581406 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:11:15.582058 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.582076 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.582427 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.582710 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.583576 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.584645 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.585775 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.586329 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.586472 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.589164 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.589168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.589190 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.589216 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.590814 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.591272 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.591277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.591281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.591424 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.591678 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.591680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.591681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.592071 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.592222 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.592481 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.592743 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":110,"threads_started":1}
I20260924 07:11:15.593602 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:15.593649 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f78064a58c84d87a7a6db20d7392b3a, State: Initialized, Role: FOLLOWER
I20260924 07:11:15.594079 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260924 07:11:15.594206 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:15.594290 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:15.594388 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:15.595274 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260924 07:11:15.595665 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f78064a58c84d87a7a6db20d7392b3a; no voters:
I20260924 07:11:15.595997 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:15.596095 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:15.596302 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [term 1 LEADER]: Becoming Leader. State: Replica: 0f78064a58c84d87a7a6db20d7392b3a, State: Running, Role: LEADER
I20260924 07:11:15.596629 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } }
I20260924 07:11:15.597168 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:15.598192 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f78064a58c84d87a7a6db20d7392b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } } }
I20260924 07:11:15.598230 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f78064a58c84d87a7a6db20d7392b3a. Latest consensus state: current_term: 1 leader_uuid: "0f78064a58c84d87a7a6db20d7392b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f78064a58c84d87a7a6db20d7392b3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41789 } } }
I20260924 07:11:15.598313 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:15.598325 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:15.598680 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:15.601861 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:15.607581 1571 catalog_manager.cc:1347] Generated new cluster ID: 324804b629754a82a36cc13b157309dd
I20260924 07:11:15.607648 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:15.626580 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:15.627588 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:15.641113 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Generated new TSK 0
I20260924 07:11:15.642292 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:15.680670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:15.793357 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:15.793639 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:15.793684 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:15.797849 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:15.797909 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:15.797948 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:15.797987 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:15.803424 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:15.804553 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:15.805542 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:15.815294 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:15.815294 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:15.815554 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:15.815805 1583 server_base.cc:1034] running on GCE node
I20260924 07:11:15.816244 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:15.816928 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:15.818121 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790233875818090 us; error 49 us; skew 500 ppm
Sep 24 07:11:15 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:15.821094 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:15.822052 1583 webserver.cc:460] Webserver started at http://127.0.98.1:46527/ using document root <none> and password file <none>
I20260924 07:11:15.822635 1583 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:15.822705 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:15.822912 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:15.825135 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4e833c70c8f467cb64e4d8bce786a50"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "752e4b3d470b0648248a80b2a656d803"
server_key_iv: "cff88ee03285e8d87707ee9b31e31f97"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.825702 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4e833c70c8f467cb64e4d8bce786a50"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "752e4b3d470b0648248a80b2a656d803"
server_key_iv: "cff88ee03285e8d87707ee9b31e31f97"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.830104 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:11:15.832881 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.834316 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:15.834457 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "c4e833c70c8f467cb64e4d8bce786a50"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-cnv3"
server_key: "752e4b3d470b0648248a80b2a656d803"
server_key_iv: "cff88ee03285e8d87707ee9b31e31f97"
server_key_version: "encryptionkey@0"
I20260924 07:11:15.834584 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:15.850863 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:15.853907 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:15.854089 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:15.855620 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:15.855671 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.855726 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:15.855751 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:15 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:15.866262 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41767
I20260924 07:11:15.866639 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41767 every 8 connection(s)
I20260924 07:11:15.867280 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:15.869472 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.856439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41277 (local address 127.0.98.62:41789)
0924 07:11:15.856588 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.856594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.857595 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 07:11:15.858901 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.858905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.858966 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.859077 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.860714 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.861249 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.862775 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.863013 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.863109 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.865879 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.865884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.865886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.865917 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.867502 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.868106 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.868109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.868110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.868189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.868481 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.868484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.868485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.868653 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.868721 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.869237 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.869349 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:15.870196 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.856920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.1:41277)
0924 07:11:15.857391 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.857473 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.858575 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.859278 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.859288 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.859804 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:15.860562 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.860584 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.861391 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.861395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.862644 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.862655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.863164 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.863168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.863351 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.864009 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:15.864025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:15.865659 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:15.867614 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.867619 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.867635 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.867981 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.868275 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.868278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.868279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.868414 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.868735 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:15.868740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:15.868978 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.869149 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.869406 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260924 07:11:15.871515 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260924 07:11:15.871657 1714 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:15.872242 1714 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
I20260924 07:11:15.873997 1510 ts_manager.cc:194] Registered new tserver with Master: c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767)
I20260924 07:11:15.875351 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41277
I20260924 07:11:15.877108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1583
I20260924 07:11:15.877223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:15.877521 392 external_mini_cluster.cc:1428] Setting key 5f0461176d212c620ea0aa988c7cf229
I20260924 07:11:15.881242 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.873852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.1:50529)
0924 07:11:15.874143 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.874158 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.874258 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.874484 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.874487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.874695 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:15.874814 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.874822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.875616 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.875619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.876275 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.876284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.876721 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.876725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.876805 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.877344 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:15.877359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:15.877800 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:15.879675 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.879679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.879681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.879917 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.880211 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.880214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.880216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.880304 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.880750 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:15.880753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:15.880841 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.880943 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.881074 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260924 07:11:15.881268 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.873965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50529 (local address 127.0.98.62:41789)
0924 07:11:15.874084 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.874088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.874227 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:11:15.874335 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.874338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.874377 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.874443 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.874923 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.875475 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.876381 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.876591 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.876674 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.878166 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.878170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.878172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.878194 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.879574 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.880063 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.880065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.880067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.880138 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.880402 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.880406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.880407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.880635 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.880734 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.881002 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.881118 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:15.902662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:16.011747 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:16.012020 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:16.012075 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:16.016431 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:16.016516 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:16.016559 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:16.016592 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:16.022135 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:16.023267 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:16.024322 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:16.035126 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:16.035194 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:16.035181 1723 server_base.cc:1034] running on GCE node
W20260924 07:11:16.035324 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:16.035772 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:16.036487 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:16.037678 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790233876037654 us; error 37 us; skew 500 ppm
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:16.040840 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:16.041817 1723 webserver.cc:460] Webserver started at http://127.0.98.2:39053/ using document root <none> and password file <none>
I20260924 07:11:16.042377 1723 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:16.042445 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:16.042663 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:16.044835 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f656af9c55a46ccaa7a6fd197cca0f3"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "d93cd860429e808d0fb3c9016f133346"
server_key_iv: "b0ebc957b2624c6b68716f0132835466"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.045408 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f656af9c55a46ccaa7a6fd197cca0f3"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "d93cd860429e808d0fb3c9016f133346"
server_key_iv: "b0ebc957b2624c6b68716f0132835466"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.049191 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:16.051812 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:16.053133 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:16.053251 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "7f656af9c55a46ccaa7a6fd197cca0f3"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "d93cd860429e808d0fb3c9016f133346"
server_key_iv: "b0ebc957b2624c6b68716f0132835466"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.053349 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:16.070380 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:16.073881 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:16.074077 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:16.075626 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:16.075688 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:16.075744 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:16.075767 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:16.085553 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40217
I20260924 07:11:16.085664 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40217 every 8 connection(s)
I20260924 07:11:16.086580 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:16.088934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1723
I20260924 07:11:16.089054 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:16.089053 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.076483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60379 (local address 127.0.98.62:41789)
0924 07:11:16.076624 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.076629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.077509 (+ 880us) server_negotiation.cc:407] Connection header received
0924 07:11:16.078652 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.078657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.078719 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.078773 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.080311 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.081017 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.082217 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.082443 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.082532 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.085349 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.085352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.085355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.085385 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.086868 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.087495 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.087499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.087501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.087597 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.087959 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.087962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.087964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.088162 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.088302 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.088790 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.088917 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:11:16.089329 392 external_mini_cluster.cc:1428] Setting key f316f24a68b4aaa72599e32b4539196c
I20260924 07:11:16.090081 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.076899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.2:60379)
0924 07:11:16.077342 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.077393 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.078374 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.079207 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.079214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.079646 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:16.080161 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.080176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.081137 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.081141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.082096 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.082112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.082603 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.082607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.082771 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.083318 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:16.083334 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:16.085137 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:16.086977 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.086983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.086998 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.087339 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.087683 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.087685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.087687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.087820 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.088308 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:16.088313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:16.088566 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.088742 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.089101 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":96,"threads_started":1}
I20260924 07:11:16.091248 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260924 07:11:16.091430 1854 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:16.091912 1854 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
I20260924 07:11:16.092998 1510 ts_manager.cc:194] Registered new tserver with Master: 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217)
I20260924 07:11:16.093741 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60379
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:16.102408 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.094265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38885 (local address 127.0.98.62:41789)
0924 07:11:16.094374 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.094378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.095858 (+ 1480us) server_negotiation.cc:407] Connection header received
0924 07:11:16.095961 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.095963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.096002 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.096081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.096559 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.097054 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.097989 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.098286 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.098369 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.099544 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.099548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.099550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.099574 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.100857 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.101233 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.101236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.101238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.101312 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.101544 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.101547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.101549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.101788 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.101875 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.102125 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.102255 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:11:16.102415 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.094225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.2:38885)
0924 07:11:16.095738 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.095754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.095875 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.096109 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.096112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.096331 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:16.096456 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.096463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.097225 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.097227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.097892 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.097901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.098393 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.098396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.098484 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.099049 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:16.099063 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:16.099435 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:16.100957 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.100960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.100963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.101149 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.101389 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.101392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.101394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.101470 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.101887 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:16.101890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:16.101978 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.102060 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.102209 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":1289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:16.119287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41789
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:16.229566 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:16.229827 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:16.229873 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:16.234450 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:16.234524 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:16.234567 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:16.234611 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:16.240198 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:16.241225 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:16.242214 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:16.251718 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:16.251763 1863 server_base.cc:1034] running on GCE node
W20260924 07:11:16.251955 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:16.251940 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:16.252482 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:16.253125 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:16.254285 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790233876254266 us; error 32 us; skew 500 ppm
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:16.257159 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:16.258114 1863 webserver.cc:460] Webserver started at http://127.0.98.3:44455/ using document root <none> and password file <none>
I20260924 07:11:16.258644 1863 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:16.258711 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:16.258899 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:16.260982 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "53a3021bdc634064bd5d32dcce7db9e4"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "ee29e0a3b2a4416beea5002e6c956370"
server_key_iv: "8ecbc359fbbbecb37a1e053d926e845d"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.261504 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "53a3021bdc634064bd5d32dcce7db9e4"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "ee29e0a3b2a4416beea5002e6c956370"
server_key_iv: "8ecbc359fbbbecb37a1e053d926e845d"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.265236 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:11:16.267659 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:16.268936 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:16.269042 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "53a3021bdc634064bd5d32dcce7db9e4"
format_stamp: "Formatted at 2026-09-24 07:11:16 on dist-test-slave-cnv3"
server_key: "ee29e0a3b2a4416beea5002e6c956370"
server_key_iv: "8ecbc359fbbbecb37a1e053d926e845d"
server_key_version: "encryptionkey@0"
I20260924 07:11:16.269137 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:16.298888 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:16.302222 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:16.302402 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:16.303902 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:16.303954 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:16.304006 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:16.304034 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:16.313908 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40085
I20260924 07:11:16.313971 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40085 every 8 connection(s)
I20260924 07:11:16.314877 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:16.315675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 1863
I20260924 07:11:16.315800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:16.316036 392 external_mini_cluster.cc:1428] Setting key c403ca89988e6b41c48f2a0446bf495a
I20260924 07:11:16.317342 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.304614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47967 (local address 127.0.98.62:41789)
0924 07:11:16.304729 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.304732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.305788 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 07:11:16.306815 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.306819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.306880 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.306934 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.308559 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.309204 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.310382 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.310606 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.310693 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.313420 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.313424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.313426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.313452 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.314994 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.315637 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.315641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.315643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.315733 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.316097 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.316100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.316102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.316307 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.316393 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.317003 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.317136 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:11:16.318243 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.305185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.3:47967)
0924 07:11:16.305646 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.305688 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.306566 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.307358 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.307368 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.307783 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:16.308419 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.308433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.309313 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.309317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.310275 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.310283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.310750 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.310754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.310915 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.311476 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:16.311491 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:16.313236 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:16.315121 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.315126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.315143 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.315511 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.315825 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.315828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.315830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.315990 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.316406 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:16.316413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:16.316722 (+ 309us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.316911 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.317230 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260924 07:11:16.319422 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41789
I20260924 07:11:16.319576 1994 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:16.320218 1994 heartbeater.cc:511] Master 127.0.98.62:41789 requested a full tablet report, sending...
I20260924 07:11:16.321215 1511 ts_manager.cc:194] Registered new tserver with Master: 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.321943 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47967
I20260924 07:11:16.328219 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.321644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.98.3:56475)
0924 07:11:16.321959 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.321971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.322061 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.322280 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.322282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.322484 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:16.322646 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.322654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.323362 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.323365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.324115 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.324124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.324516 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.324518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.324580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.325033 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:16.325043 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:16.325366 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:16.326780 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.326782 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.326784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.326957 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.327185 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.327187 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.327189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.327284 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.327773 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:16.327776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:16.327857 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.327942 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.328053 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260924 07:11:16.328253 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.321730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56475 (local address 127.0.98.62:41789)
0924 07:11:16.321835 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.321838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.322038 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:11:16.322149 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.322152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.322193 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.322254 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.322753 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.323245 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.324220 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.324433 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.324507 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.325490 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.325493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.325495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.325516 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.326687 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.327025 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.327028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.327029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.327091 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.327434 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.327439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.327440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.327676 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.327782 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.328022 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.328122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:11:16.329847 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:16.353348 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.345842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.0.98.62:41789)
0924 07:11:16.345965 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.345968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.345979 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:16.346019 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.346021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.346057 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.346118 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.346834 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.347367 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.348276 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.348491 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.348570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.350804 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.350807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.350809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.350834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.352114 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.352451 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.352453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.352455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.352518 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.352696 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.352698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.352699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.352853 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.352945 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.353105 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.353228 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:11:16.356035 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:16.359215 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:16.373514 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.368098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40217 (local address 127.0.0.1:35748)
0924 07:11:16.368421 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.368433 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.368501 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.369384 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.369387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.369404 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.369532 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.369538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370870 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.370873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.372091 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.372100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.373102 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.373106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.373225 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.373245 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.373324 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.373384 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":82,"thread_start_us":86,"threads_started":1}
I20260924 07:11:16.373538 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.367421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.0.1:35998)
0924 07:11:16.368160 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.368175 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.368256 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.369323 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.369326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.369343 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.369486 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.369494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370898 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.370901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.372037 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.372047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.373094 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.373098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.373230 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.373254 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.373316 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.373389 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":82,"thread_start_us":105,"threads_started":1}
I20260924 07:11:16.373572 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.367512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.98.1:41767)
0924 07:11:16.367881 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.367895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.367918 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:11:16.368141 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.368148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.368475 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.368618 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.369451 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370252 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.371962 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.372640 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.372778 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.373024 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.373364 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.373462 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":91,"threads_started":1}
I20260924 07:11:16.373687 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.368479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35748 (local address 127.0.98.2:40217)
0924 07:11:16.369029 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.369037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.369055 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:11:16.369113 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.369117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.369252 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.369391 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.369626 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370753 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.372225 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.372979 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.373099 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.373264 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.373454 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.373547 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":99,"threads_started":1}
I20260924 07:11:16.373370 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.367346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41767 (local address 127.0.0.1:46180)
0924 07:11:16.367783 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.367829 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.367945 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.368651 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.368657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.368679 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.368797 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.368802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370356 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.370359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.371842 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.371853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.372788 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.372791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.372946 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.372987 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.373101 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.373221 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":19,"thread_start_us":66,"threads_started":1}
I20260924 07:11:16.374289 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.368146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35998 (local address 127.0.98.3:40085)
0924 07:11:16.368952 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.368962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.368984 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:11:16.369050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.369054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.369201 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.369310 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.369602 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.370774 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.372167 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.372952 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.373882 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.374031 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.374078 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.374155 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":97,"threads_started":1}
I20260924 07:11:16.376912 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d9d320fd909443928eb7db6fba092c24 (DEFAULT_TABLE table=test-table [id=fd934b7a53a24943a663e30584a5f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.376984 1928 tablet_service.cc:1467] Processing CreateTablet for tablet d9d320fd909443928eb7db6fba092c24 (DEFAULT_TABLE table=test-table [id=fd934b7a53a24943a663e30584a5f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.376982 1788 tablet_service.cc:1467] Processing CreateTablet for tablet d9d320fd909443928eb7db6fba092c24 (DEFAULT_TABLE table=test-table [id=fd934b7a53a24943a663e30584a5f956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.377838 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d320fd909443928eb7db6fba092c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.377983 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d320fd909443928eb7db6fba092c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.377990 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9d320fd909443928eb7db6fba092c24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.383920 2017 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Bootstrap starting.
I20260924 07:11:16.385317 2018 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Bootstrap starting.
I20260924 07:11:16.386096 2019 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Bootstrap starting.
I20260924 07:11:16.386593 2017 tablet_bootstrap.cc:654] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.387658 2018 tablet_bootstrap.cc:654] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.387816 2019 tablet_bootstrap.cc:654] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.387902 2017 log.cc:824] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:16.388481 2018 log.cc:824] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:16.388588 2019 log.cc:824] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:16.389818 2017 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: No bootstrap required, opened a new log
I20260924 07:11:16.390005 2017 ts_tablet_manager.cc:1397] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:16.390381 2019 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: No bootstrap required, opened a new log
I20260924 07:11:16.390381 2018 tablet_bootstrap.cc:492] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: No bootstrap required, opened a new log
I20260924 07:11:16.390563 2018 ts_tablet_manager.cc:1397] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:11:16.390563 2019 ts_tablet_manager.cc:1397] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:11:16.393204 2017 raft_consensus.cc:348] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.393473 2017 raft_consensus.cc:374] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.393529 2017 raft_consensus.cc:729] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e833c70c8f467cb64e4d8bce786a50, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.394045 2017 consensus_queue.cc:260] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.394359 2018 raft_consensus.cc:348] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.394344 2019 raft_consensus.cc:348] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.394601 2019 raft_consensus.cc:374] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.394601 2018 raft_consensus.cc:374] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.394652 2019 raft_consensus.cc:729] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a3021bdc634064bd5d32dcce7db9e4, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.394658 2018 raft_consensus.cc:729] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f656af9c55a46ccaa7a6fd197cca0f3, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.395018 2017 ts_tablet_manager.cc:1428] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 07:11:16.395082 2018 consensus_queue.cc:260] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.395082 2019 consensus_queue.cc:260] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.395352 1714 heartbeater.cc:503] Master 127.0.98.62:41789 was elected leader, sending a full tablet report...
I20260924 07:11:16.396050 2019 ts_tablet_manager.cc:1428] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 07:11:16.396050 2018 ts_tablet_manager.cc:1428] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 07:11:16.396380 1854 heartbeater.cc:503] Master 127.0.98.62:41789 was elected leader, sending a full tablet report...
I20260924 07:11:16.397256 1994 heartbeater.cc:503] Master 127.0.98.62:41789 was elected leader, sending a full tablet report...
W20260924 07:11:16.565958 1995 tablet.cc:2369] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:16.587882 1855 tablet.cc:2369] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:16.618742 1715 tablet.cc:2369] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:16.698951 2023 raft_consensus.cc:482] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:16.699150 2023 raft_consensus.cc:504] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.700415 2023 leader_election.cc:288] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085), 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217)
I20260924 07:11:16.703357 2024 raft_consensus.cc:482] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:16.703589 2024 raft_consensus.cc:504] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.704558 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.700882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60099 (local address 127.0.98.2:40217)
0924 07:11:16.701013 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.701018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.701234 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:11:16.701369 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.701372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.701416 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.701477 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.701748 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.702262 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.703440 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.703970 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.704068 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.704131 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.704345 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.704430 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:11:16.704633 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.700757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.98.1:33079)
0924 07:11:16.701118 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.701130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.701253 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.701477 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.701480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.701494 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.701626 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.701631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.702335 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.702338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.703414 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.703426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.704247 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.704251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.704349 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.704367 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.704427 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.704494 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":24,"spinlock_wait_cycles":23936,"thread_start_us":80,"threads_started":1}
I20260924 07:11:16.704653 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.700875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33079 (local address 127.0.98.3:40085)
0924 07:11:16.701073 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.701077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.701208 (+ 131us) server_negotiation.cc:407] Connection header received
0924 07:11:16.701351 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.701354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.701395 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.701453 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.701729 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.702231 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.703570 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.704121 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.704240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.704305 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.704453 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.704531 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 07:11:16.704957 2024 leader_election.cc:288] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767), 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.705334 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.700745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40217 (local address 127.0.98.1:60099)
0924 07:11:16.701074 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.701088 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.701273 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.701492 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.701494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.701543 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.701649 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.701654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.702366 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.702369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.703291 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.703304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.704090 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.704094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.704197 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.704221 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.705102 (+ 881us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.705191 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":47,"spinlock_wait_cycles":6528,"thread_start_us":107,"threads_started":1}
I20260924 07:11:16.705278 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4" is_pre_election: true
I20260924 07:11:16.705590 1948 raft_consensus.cc:2454] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 0.
I20260924 07:11:16.706015 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" is_pre_election: true
I20260924 07:11:16.706103 1600 leader_election.cc:302] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a3021bdc634064bd5d32dcce7db9e4, c4e833c70c8f467cb64e4d8bce786a50; no voters:
I20260924 07:11:16.706224 1808 raft_consensus.cc:2454] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 0.
I20260924 07:11:16.706423 2023 raft_consensus.cc:2781] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:16.706513 2023 raft_consensus.cc:482] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:16.706558 2023 raft_consensus.cc:3037] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.707814 2023 raft_consensus.cc:504] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.708195 2023 leader_election.cc:288] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 election: Requested vote from peers 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085), 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217)
I20260924 07:11:16.708542 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3"
I20260924 07:11:16.708604 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4"
I20260924 07:11:16.708663 1808 raft_consensus.cc:3037] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.708734 1948 raft_consensus.cc:3037] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.709980 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.705537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36829 (local address 127.0.98.3:40085)
0924 07:11:16.705641 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.705644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.705862 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:11:16.706026 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.706029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.706074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.706148 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.706444 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.707045 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.708673 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.709304 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.709380 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.709439 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.709790 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.709870 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:11:16.710034 1808 raft_consensus.cc:2454] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 1.
I20260924 07:11:16.710418 1948 raft_consensus.cc:2454] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 1.
I20260924 07:11:16.710436 1601 leader_election.cc:302] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f656af9c55a46ccaa7a6fd197cca0f3, c4e833c70c8f467cb64e4d8bce786a50; no voters:
I20260924 07:11:16.710549 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.705186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40085 (local address 127.0.98.2:36829)
0924 07:11:16.705788 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.705802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.705927 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.706225 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.706228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.706246 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.706347 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.706353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.707159 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.707162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.708495 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.708505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.709533 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.709536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.709641 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.709670 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.710259 (+ 589us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.710343 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":102,"threads_started":1}
I20260924 07:11:16.710866 2023 raft_consensus.cc:2781] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:16.711088 2023 raft_consensus.cc:686] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 LEADER]: Becoming Leader. State: Replica: c4e833c70c8f467cb64e4d8bce786a50, State: Running, Role: LEADER
I20260924 07:11:16.711455 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.705633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34555 (local address 127.0.98.1:41767)
0924 07:11:16.705739 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.705743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.707674 (+ 1931us) server_negotiation.cc:407] Connection header received
0924 07:11:16.707819 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.707822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.707862 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.707918 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:16.708143 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.708799 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.710254 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.710832 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.710899 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.710975 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.711258 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.711326 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:11:16.711479 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.705279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41767 (local address 127.0.98.2:34555)
0924 07:11:16.707592 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.707605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.707728 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.707942 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.707945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.707959 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:16.708056 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.708060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.708920 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.708923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.710104 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.710114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.711062 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.711065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.711137 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.711169 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.711244 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.711309 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"thread_start_us":108,"threads_started":1}
I20260924 07:11:16.711454 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4" is_pre_election: true
I20260924 07:11:16.711628 1948 raft_consensus.cc:2379] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f656af9c55a46ccaa7a6fd197cca0f3 in current term 1: Already voted for candidate c4e833c70c8f467cb64e4d8bce786a50 in this term.
I20260924 07:11:16.711943 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9d320fd909443928eb7db6fba092c24" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e833c70c8f467cb64e4d8bce786a50" is_pre_election: true
I20260924 07:11:16.712190 2023 consensus_queue.cc:237] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } }
I20260924 07:11:16.712510 1741 leader_election.cc:302] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f656af9c55a46ccaa7a6fd197cca0f3; no voters: 53a3021bdc634064bd5d32dcce7db9e4, c4e833c70c8f467cb64e4d8bce786a50
I20260924 07:11:16.712731 2024 raft_consensus.cc:2726] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 07:11:16.715404 1509 catalog_manager.cc:5495] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:16.745527 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.741827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.98.1:41767)
0924 07:11:16.741958 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.741962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.741979 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:11:16.742164 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.742168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.742241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.742328 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:16.742585 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.743093 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.744052 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.744269 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.744355 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.744706 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:16.745139 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:16.745241 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.745337 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.745393 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:16.751417 1948 raft_consensus.cc:1261] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Refusing update from remote peer c4e833c70c8f467cb64e4d8bce786a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:16.751474 1808 raft_consensus.cc:1261] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Refusing update from remote peer c4e833c70c8f467cb64e4d8bce786a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:16.752180 2030 consensus_queue.cc:1035] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:16.752379 2023 consensus_queue.cc:1035] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:16.758193 2033 mvcc.cc:205] Tried to move back new op lower bound from 7332797959165874176 to 7332797959015739392. Current Snapshot: MvccSnapshot[applied={T|T < 7332797959165874176}]
I20260924 07:11:16.760130 2034 mvcc.cc:205] Tried to move back new op lower bound from 7332797959165874176 to 7332797959015739392. Current Snapshot: MvccSnapshot[applied={T|T < 7332797959165874176}]
I20260924 07:11:16.770533 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:16.770721 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:16.773542 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Sending DeleteTablet for 3 replicas of tablet d9d320fd909443928eb7db6fba092c24
I20260924 07:11:16.774408 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d320fd909443928eb7db6fba092c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46180
I20260924 07:11:16.774755 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d320fd909443928eb7db6fba092c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35748
I20260924 07:11:16.774899 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet d9d320fd909443928eb7db6fba092c24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35998
I20260924 07:11:16.775309 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:11:16.775653 2042 tablet_replica.cc:330] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: stopping tablet replica
W20260924 07:11:16.775767 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:16.775987 2042 raft_consensus.cc:2229] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:16.776578 2043 tablet_replica.cc:330] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: stopping tablet replica
I20260924 07:11:16.776875 2043 raft_consensus.cc:2229] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:16.776368 2042 raft_consensus.cc:2258] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:16.777190 2043 raft_consensus.cc:2258] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:16.777704 2044 tablet_replica.cc:330] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: stopping tablet replica
I20260924 07:11:16.777982 2044 raft_consensus.cc:2229] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:16.778285 2044 raft_consensus.cc:2258] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:16.779137 2043 ts_tablet_manager.cc:1905] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:16.780151 2044 ts_tablet_manager.cc:1905] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:16.781518 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d (DEFAULT_TABLE table=test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.781791 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7699920814034a6dbe7b8fa834ba5b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.781795 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d (DEFAULT_TABLE table=test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.782024 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7699920814034a6dbe7b8fa834ba5b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.782766 2044 ts_tablet_manager.cc:1918] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:16.782848 2044 log.cc:1196] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/d9d320fd909443928eb7db6fba092c24
I20260924 07:11:16.783238 2044 ts_tablet_manager.cc:1939] T d9d320fd909443928eb7db6fba092c24 P 53a3021bdc634064bd5d32dcce7db9e4: Deleting consensus metadata
I20260924 07:11:16.783653 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d (DEFAULT_TABLE table=test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:16.783890 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7699920814034a6dbe7b8fa834ba5b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.784554 1496 catalog_manager.cc:4841] TS 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085): tablet d9d320fd909443928eb7db6fba092c24 (table test-table [id=fd934b7a53a24943a663e30584a5f956]) successfully deleted
I20260924 07:11:16.784996 2018 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Bootstrap starting.
I20260924 07:11:16.785845 2019 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Bootstrap starting.
I20260924 07:11:16.785931 2018 tablet_bootstrap.cc:654] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.786712 2019 tablet_bootstrap.cc:654] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.786980 2017 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Bootstrap starting.
I20260924 07:11:16.787866 2042 ts_tablet_manager.cc:1905] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:16.787935 2017 tablet_bootstrap.cc:654] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.788233 2043 ts_tablet_manager.cc:1918] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:16.788385 2043 log.cc:1196] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/d9d320fd909443928eb7db6fba092c24
I20260924 07:11:16.788733 2043 ts_tablet_manager.cc:1939] T d9d320fd909443928eb7db6fba092c24 P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting consensus metadata
I20260924 07:11:16.788906 2019 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: No bootstrap required, opened a new log
I20260924 07:11:16.789003 2019 ts_tablet_manager.cc:1397] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:16.789485 2019 raft_consensus.cc:348] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.789602 2019 raft_consensus.cc:374] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.789636 2019 raft_consensus.cc:729] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a3021bdc634064bd5d32dcce7db9e4, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.789750 2019 consensus_queue.cc:260] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.790015 2019 ts_tablet_manager.cc:1428] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:16.790381 2042 ts_tablet_manager.cc:1918] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:16.790475 2042 log.cc:1196] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/d9d320fd909443928eb7db6fba092c24
I20260924 07:11:16.790817 2042 ts_tablet_manager.cc:1939] T d9d320fd909443928eb7db6fba092c24 P c4e833c70c8f467cb64e4d8bce786a50: Deleting consensus metadata
I20260924 07:11:16.791457 1497 catalog_manager.cc:4841] TS 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217): tablet d9d320fd909443928eb7db6fba092c24 (table test-table [id=fd934b7a53a24943a663e30584a5f956]) successfully deleted
I20260924 07:11:16.791960 1497 catalog_manager.cc:4841] TS c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767): tablet d9d320fd909443928eb7db6fba092c24 (table test-table [id=fd934b7a53a24943a663e30584a5f956]) successfully deleted
I20260924 07:11:16.792773 2018 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: No bootstrap required, opened a new log
I20260924 07:11:16.792855 2018 ts_tablet_manager.cc:1397] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260924 07:11:16.793229 2018 raft_consensus.cc:348] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.793321 2018 raft_consensus.cc:374] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.793349 2018 raft_consensus.cc:729] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f656af9c55a46ccaa7a6fd197cca0f3, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.793469 2018 consensus_queue.cc:260] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.793709 2018 ts_tablet_manager.cc:1428] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:16.796113 2017 tablet_bootstrap.cc:492] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: No bootstrap required, opened a new log
I20260924 07:11:16.796204 2017 ts_tablet_manager.cc:1397] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 07:11:16.796640 2017 raft_consensus.cc:348] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.796793 2017 raft_consensus.cc:374] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.796826 2017 raft_consensus.cc:729] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e833c70c8f467cb64e4d8bce786a50, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.796950 2017 consensus_queue.cc:260] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.797216 2017 ts_tablet_manager.cc:1428] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:16.824805 2024 raft_consensus.cc:482] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:16.824954 2024 raft_consensus.cc:504] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.825387 2024 leader_election.cc:288] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767), 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.825842 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7699920814034a6dbe7b8fa834ba5b2d" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e833c70c8f467cb64e4d8bce786a50" is_pre_election: true
I20260924 07:11:16.825881 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7699920814034a6dbe7b8fa834ba5b2d" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4" is_pre_election: true
I20260924 07:11:16.825999 1948 raft_consensus.cc:2454] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f656af9c55a46ccaa7a6fd197cca0f3 in term 0.
I20260924 07:11:16.826040 1668 raft_consensus.cc:2454] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f656af9c55a46ccaa7a6fd197cca0f3 in term 0.
I20260924 07:11:16.826290 1740 leader_election.cc:302] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a3021bdc634064bd5d32dcce7db9e4, 7f656af9c55a46ccaa7a6fd197cca0f3; no voters:
I20260924 07:11:16.826484 2024 raft_consensus.cc:2781] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:16.826540 2024 raft_consensus.cc:482] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:16.826577 2024 raft_consensus.cc:3037] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.828082 2024 raft_consensus.cc:504] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.828423 2024 leader_election.cc:288] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 election: Requested vote from peers c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767), 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.828824 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7699920814034a6dbe7b8fa834ba5b2d" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4e833c70c8f467cb64e4d8bce786a50"
I20260924 07:11:16.828824 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7699920814034a6dbe7b8fa834ba5b2d" candidate_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4"
I20260924 07:11:16.828953 1948 raft_consensus.cc:3037] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.828953 1668 raft_consensus.cc:3037] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.830061 1668 raft_consensus.cc:2454] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f656af9c55a46ccaa7a6fd197cca0f3 in term 1.
I20260924 07:11:16.830062 1948 raft_consensus.cc:2454] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f656af9c55a46ccaa7a6fd197cca0f3 in term 1.
I20260924 07:11:16.830364 1740 leader_election.cc:302] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a3021bdc634064bd5d32dcce7db9e4, 7f656af9c55a46ccaa7a6fd197cca0f3; no voters:
I20260924 07:11:16.830578 2024 raft_consensus.cc:2781] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:16.830778 2024 raft_consensus.cc:686] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 LEADER]: Becoming Leader. State: Replica: 7f656af9c55a46ccaa7a6fd197cca0f3, State: Running, Role: LEADER
I20260924 07:11:16.831080 2024 consensus_queue.cc:237] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.833660 1510 catalog_manager.cc:5495] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:11:16 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:16.889847 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.881204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41789 (local address 127.0.0.1:39752)
0924 07:11:16.881522 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.881537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.881652 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.881877 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.881879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.882261 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:16.882357 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.882364 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.883079 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.883082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.883670 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.883678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.884082 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.884085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.884200 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.884893 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:16.884905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:16.886525 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:16.888255 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.888267 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.888274 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.888501 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.888754 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:16.888756 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:16.888759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:16.888831 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:16.889248 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:16.889255 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:16.889363 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.889457 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.889616 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":33,"thread_start_us":80,"threads_started":1}
I20260924 07:11:16.889847 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.881271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.0.98.62:41789)
0924 07:11:16.881512 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.881516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.881615 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:11:16.881731 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.881734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.881778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.881848 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:16.882466 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.882984 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.883777 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.883988 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.884069 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.886682 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:16.886685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:16.886687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:16.886707 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:16.888141 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.888593 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.888597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.888598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.888674 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:16.888906 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:16.888909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:16.888911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:16.889087 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:16.889251 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.889513 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.889666 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":190,"thread_start_us":56,"threads_started":1}
I20260924 07:11:16.895021 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:11:16.895953 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:16.901151 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 75d0596b7bb6453184410831bff16674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bfc9bfe6a6648b9aa9383a7f0901e01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:16.901219 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 75d0596b7bb6453184410831bff16674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bfc9bfe6a6648b9aa9383a7f0901e01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:16.901569 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d0596b7bb6453184410831bff16674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.901576 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d0596b7bb6453184410831bff16674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.903972 2019 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: Bootstrap starting.
I20260924 07:11:16.904644 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 75d0596b7bb6453184410831bff16674 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2bfc9bfe6a6648b9aa9383a7f0901e01]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:16.904917 2019 tablet_bootstrap.cc:654] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.905025 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75d0596b7bb6453184410831bff16674. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:16.905902 2019 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: No bootstrap required, opened a new log
I20260924 07:11:16.905979 2019 ts_tablet_manager.cc:1397] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:16.906335 2019 raft_consensus.cc:348] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.906430 2019 raft_consensus.cc:374] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.906461 2019 raft_consensus.cc:729] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a3021bdc634064bd5d32dcce7db9e4, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.906575 2019 consensus_queue.cc:260] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.906841 2019 ts_tablet_manager.cc:1428] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:16.907079 2025 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.907217 2025 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.907554 2017 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: Bootstrap starting.
I20260924 07:11:16.908511 2017 tablet_bootstrap.cc:654] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.909868 2017 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: No bootstrap required, opened a new log
I20260924 07:11:16.909964 2017 ts_tablet_manager.cc:1397] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:16.910400 2017 raft_consensus.cc:348] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.910519 2017 raft_consensus.cc:374] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.910564 2017 raft_consensus.cc:729] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e833c70c8f467cb64e4d8bce786a50, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.910686 2017 consensus_queue.cc:260] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.910956 2017 ts_tablet_manager.cc:1428] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:16.911139 2030 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.911271 2030 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.911512 2018 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: Bootstrap starting.
I20260924 07:11:16.912496 2018 tablet_bootstrap.cc:654] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:16.914279 2018 tablet_bootstrap.cc:492] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: No bootstrap required, opened a new log
I20260924 07:11:16.914363 2018 ts_tablet_manager.cc:1397] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 07:11:16.914743 2018 raft_consensus.cc:348] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.914852 2018 raft_consensus.cc:374] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:16.914880 2018 raft_consensus.cc:729] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f656af9c55a46ccaa7a6fd197cca0f3, State: Initialized, Role: FOLLOWER
I20260924 07:11:16.915010 2018 consensus_queue.cc:260] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.915283 2018 ts_tablet_manager.cc:1428] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:16.915238 2024 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.915369 2024 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.949249 2030 raft_consensus.cc:482] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:16.949397 2030 raft_consensus.cc:504] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.949805 2030 leader_election.cc:288] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217), 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.950274 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d0596b7bb6453184410831bff16674" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4" is_pre_election: true
I20260924 07:11:16.950284 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d0596b7bb6453184410831bff16674" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" is_pre_election: true
I20260924 07:11:16.950408 1807 raft_consensus.cc:2454] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 0.
I20260924 07:11:16.950418 1948 raft_consensus.cc:2454] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 0.
I20260924 07:11:16.950700 1600 leader_election.cc:302] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a3021bdc634064bd5d32dcce7db9e4, c4e833c70c8f467cb64e4d8bce786a50; no voters:
I20260924 07:11:16.950903 2030 raft_consensus.cc:2781] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:16.950965 2030 raft_consensus.cc:482] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:16.950997 2030 raft_consensus.cc:3037] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.952100 2030 raft_consensus.cc:504] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.952435 2030 leader_election.cc:288] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 election: Requested vote from peers 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217), 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085)
I20260924 07:11:16.952785 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d0596b7bb6453184410831bff16674" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53a3021bdc634064bd5d32dcce7db9e4"
I20260924 07:11:16.952806 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75d0596b7bb6453184410831bff16674" candidate_uuid: "c4e833c70c8f467cb64e4d8bce786a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3"
I20260924 07:11:16.952900 1948 raft_consensus.cc:3037] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.952927 1807 raft_consensus.cc:3037] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:16.953845 1948 raft_consensus.cc:2454] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 1.
I20260924 07:11:16.954020 1807 raft_consensus.cc:2454] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e833c70c8f467cb64e4d8bce786a50 in term 1.
I20260924 07:11:16.954134 1600 leader_election.cc:302] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a3021bdc634064bd5d32dcce7db9e4, c4e833c70c8f467cb64e4d8bce786a50; no voters:
I20260924 07:11:16.954381 2030 raft_consensus.cc:2781] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:16.954486 2030 raft_consensus.cc:686] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [term 1 LEADER]: Becoming Leader. State: Replica: c4e833c70c8f467cb64e4d8bce786a50, State: Running, Role: LEADER
I20260924 07:11:16.954633 2030 consensus_queue.cc:237] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } }
I20260924 07:11:16.955400 2023 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: TxnStatusTablet state changed. Reason: New leader c4e833c70c8f467cb64e4d8bce786a50. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.955505 2023 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:16.955789 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:11:16.955961 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:11:16.956511 1510 catalog_manager.cc:5495] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:16.975266 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.971582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41767 (local address 127.0.0.1:46200)
0924 07:11:16.971770 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.971786 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.971881 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.972166 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.972168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.972177 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:16.972287 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.972309 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.973309 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.973314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.973997 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.974007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.974545 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.974548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:16.974637 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.974683 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:16.974991 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:16.975006 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:11:16.975072 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:11:16.975155 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 07:11:16.975282 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.971728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.98.1:41767)
0924 07:11:16.971867 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.971872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.971888 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:11:16.971976 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.971979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.972032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.972107 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:16.972389 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.973198 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.974097 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.974440 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.974539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.974806 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:16.974889 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:16.974955 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.975091 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.975145 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:16.977912 1807 raft_consensus.cc:1261] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Refusing update from remote peer c4e833c70c8f467cb64e4d8bce786a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:16.977923 1948 raft_consensus.cc:1261] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Refusing update from remote peer c4e833c70c8f467cb64e4d8bce786a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:16.978324 2023 consensus_queue.cc:1035] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:16.978477 2030 consensus_queue.cc:1035] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:16.980031 2024 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: TxnStatusTablet state changed. Reason: New leader c4e833c70c8f467cb64e4d8bce786a50. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.980170 2024 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.981786 2025 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: TxnStatusTablet state changed. Reason: New leader c4e833c70c8f467cb64e4d8bce786a50. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.981892 2025 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.982013 2030 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.982124 2030 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:16.982587 2024 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.982702 2024 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 7f656af9c55a46ccaa7a6fd197cca0f3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.982740 2030 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.982838 2030 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P c4e833c70c8f467cb64e4d8bce786a50: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:16.983942 2035 mvcc.cc:205] Tried to move back new op lower bound from 7332797960097538048 to 7332797960007258112. Current Snapshot: MvccSnapshot[applied={T|T < 7332797960097538048 or (T in {7332797960097538048})}]
I20260924 07:11:16.984345 2025 tablet_replica.cc:439] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4e833c70c8f467cb64e4d8bce786a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 } } peers { permanent_uuid: "7f656af9c55a46ccaa7a6fd197cca0f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40217 } } peers { permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 } } }
I20260924 07:11:16.984443 2025 tablet_replica.cc:442] T 75d0596b7bb6453184410831bff16674 P 53a3021bdc634064bd5d32dcce7db9e4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:16.990808 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.987616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.0.98.2:40217)
0924 07:11:16.987739 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.987742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.987752 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:11:16.987815 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.987817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.987864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.987952 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:16.988173 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.988679 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.989454 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.989683 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:16.989764 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:16.990005 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:16.990446 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:16.990547 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:16.990618 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:11:16.990678 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:11:17.001818 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.998266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48227 (local address 127.0.98.1:41767)
0924 07:11:16.998406 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:16.998410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:16.998498 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:11:16.998644 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:16.998648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:16.998701 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:16.998789 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:16.999463 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:16.999973 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.000833 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.001040 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.001122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.001394 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:17.001463 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:17.001518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:17.001616 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:11:17.001671 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:11:17.001853 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:16.998046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41767 (local address 127.0.98.2:48227)
0924 07:11:16.998418 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:16.998432 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:16.998540 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:16.998847 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:16.998850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:16.998859 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:16.999312 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:16.999320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.000077 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.000081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:17.000722 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:17.000732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.001152 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.001156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:17.001228 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.001271 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:17.001533 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:17.001539 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:17.001597 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:11:17.001687 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260924 07:11:17.009531 1948 raft_consensus.cc:1261] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Refusing update from remote peer 7f656af9c55a46ccaa7a6fd197cca0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:17.009685 1668 raft_consensus.cc:1261] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Refusing update from remote peer 7f656af9c55a46ccaa7a6fd197cca0f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:17.010011 2048 consensus_queue.cc:1035] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53a3021bdc634064bd5d32dcce7db9e4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:17.010310 2024 consensus_queue.cc:1035] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4e833c70c8f467cb64e4d8bce786a50" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:17.032251 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:17.028809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33579 (local address 127.0.98.2:40217)
0924 07:11:17.028915 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:17.028919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:17.028932 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:17.028979 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:17.028981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:17.029032 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:17.029129 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:17.029289 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.030054 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.031122 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.031459 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.031553 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.031830 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:17.031908 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:17.031965 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:17.032049 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:11:17.032102 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:11:17.032251 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:17.028454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40217 (local address 127.0.98.1:33579)
0924 07:11:17.028779 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:17.028793 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:17.028876 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:17.029122 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:17.029123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:17.029128 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:17.029218 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:17.029222 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.030163 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.030167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:17.031000 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:17.031020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.031563 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.031566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:17.031651 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.031700 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:17.031978 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:17.031984 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:17.032041 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:11:17.032124 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
I20260924 07:11:17.062896 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:17.063088 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:17.065560 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f78064a58c84d87a7a6db20d7392b3a: Sending DeleteTablet for 3 replicas of tablet 7699920814034a6dbe7b8fa834ba5b2d
I20260924 07:11:17.066421 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35748
I20260924 07:11:17.066645 2043 tablet_replica.cc:330] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: stopping tablet replica
I20260924 07:11:17.066776 2043 raft_consensus.cc:2229] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:17.066924 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35998
I20260924 07:11:17.067090 2044 tablet_replica.cc:330] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: stopping tablet replica
I20260924 07:11:17.067075 2043 raft_consensus.cc:2258] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:17.067572 2044 raft_consensus.cc:2229] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:17.067704 2044 raft_consensus.cc:2258] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:17.068256 2043 ts_tablet_manager.cc:1905] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:17.068449 2044 ts_tablet_manager.cc:1905] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:17.071345 2043 ts_tablet_manager.cc:1918] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:17.071447 2043 log.cc:1196] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/7699920814034a6dbe7b8fa834ba5b2d
I20260924 07:11:17.071429 2044 ts_tablet_manager.cc:1918] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:17.071501 2044 log.cc:1196] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/7699920814034a6dbe7b8fa834ba5b2d
I20260924 07:11:17.071801 2043 ts_tablet_manager.cc:1939] T 7699920814034a6dbe7b8fa834ba5b2d P 7f656af9c55a46ccaa7a6fd197cca0f3: Deleting consensus metadata
I20260924 07:11:17.072145 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7699920814034a6dbe7b8fa834ba5b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46180
I20260924 07:11:17.072347 2042 tablet_replica.cc:330] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: stopping tablet replica
I20260924 07:11:17.072460 2042 raft_consensus.cc:2229] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:17.072587 2042 raft_consensus.cc:2258] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:17.072628 1497 catalog_manager.cc:4841] TS 7f656af9c55a46ccaa7a6fd197cca0f3 (127.0.98.2:40217): tablet 7699920814034a6dbe7b8fa834ba5b2d (table test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]) successfully deleted
I20260924 07:11:17.074689 2042 ts_tablet_manager.cc:1905] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:17.077780 2042 ts_tablet_manager.cc:1918] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:17.077860 2042 log.cc:1196] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/7699920814034a6dbe7b8fa834ba5b2d
I20260924 07:11:17.078112 2042 ts_tablet_manager.cc:1939] T 7699920814034a6dbe7b8fa834ba5b2d P c4e833c70c8f467cb64e4d8bce786a50: Deleting consensus metadata
I20260924 07:11:17.078573 2044 ts_tablet_manager.cc:1939] T 7699920814034a6dbe7b8fa834ba5b2d P 53a3021bdc634064bd5d32dcce7db9e4: Deleting consensus metadata
I20260924 07:11:17.079502 1496 catalog_manager.cc:4841] TS 53a3021bdc634064bd5d32dcce7db9e4 (127.0.98.3:40085): tablet 7699920814034a6dbe7b8fa834ba5b2d (table test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]) successfully deleted
I20260924 07:11:17.080435 1497 catalog_manager.cc:4841] TS c4e833c70c8f467cb64e4d8bce786a50 (127.0.98.1:41767): tablet 7699920814034a6dbe7b8fa834ba5b2d (table test-table [id=b0cd949e0f9c469485bbbb8c408cbd52]) successfully deleted
Sep 24 07:11:17 dist-test-slave-cnv3 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:17.092813 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:17.074247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.0.98.1:41767)
0924 07:11:17.074367 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:17.074372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:17.074911 (+ 539us) server_negotiation.cc:407] Connection header received
0924 07:11:17.075094 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:17.075097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:17.075145 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:17.075195 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:17.076196 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.076979 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.078186 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.078539 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.078609 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.088008 (+ 9399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:17.088026 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:17.088043 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:17.088081 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:17.090554 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:17.091137 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:17.091145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:17.091146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:17.091237 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:17.091570 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:17.091573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:17.091575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:17.091900 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:17.092105 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:17.092490 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 07:11:17.092620 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 07:11:17.093909 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46202
I20260924 07:11:17.109443 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:17.099842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.0.98.62:41789)
0924 07:11:17.099970 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:17.099974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:17.100462 (+ 488us) server_negotiation.cc:407] Connection header received
0924 07:11:17.100651 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:17.100654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:17.100706 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:17.100756 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:17.101645 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.102495 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.103628 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:17.103995 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:17.104067 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:17.105915 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:17.105919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:17.105922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:17.105958 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:17.107681 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:17.108126 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:17.108130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:17.108131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:17.108225 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:17.108485 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:17.108488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:17.108490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:17.108694 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:17.108794 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:17.109122 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 07:11:17.109274 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 07:11:17.110874 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39766
I20260924 07:11:17.112533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1583
I20260924 07:11:17.121425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1723
I20260924 07:11:17.126974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1863
I20260924 07:11:17.131870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 1479
2026-09-24T07:11:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4617 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:17 dist-test-slave-cnv3 krb5kdc[2083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:17 dist-test-slave-cnv3 krb5kdc[2083](info): set up 2 sockets
Sep 24 07:11:17 dist-test-slave-cnv3 krb5kdc[2083](info): commencing operation
krb5kdc: starting...
W20260924 07:11:18.103085 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:18.227494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:18.339618 2096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:18.339929 2096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:18.339989 2096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:18.344195 2096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:18.344267 2096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:11:18.344291 2096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:18.344314 2096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:18.344336 2096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:18.344365 2096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:18.349982 2096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38137
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:18.351212 2096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:18.352283 2096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:18.360540 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:18.360587 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:18.360622 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:18.360764 2096 server_base.cc:1034] running on GCE node
I20260924 07:11:18.361464 2096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:18.362560 2096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:18.363722 2096 hybrid_clock.cc:648] HybridClock initialized: now 1790233878363692 us; error 43 us; skew 500 ppm
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:18.366415 2096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:18.367336 2096 webserver.cc:460] Webserver started at http://127.0.98.62:33773/ using document root <none> and password file <none>
I20260924 07:11:18.367902 2096 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:18.367969 2096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:18.368158 2096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:18.370157 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "d43efaf146fc44b6bf01f29a0e39911c"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "3c0f4d01a5d96fd2d146ea1a156f9b8d"
server_key_iv: "cb7aea1c3c1eba126d9f2fad4e1b7305"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.370682 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "d43efaf146fc44b6bf01f29a0e39911c"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "3c0f4d01a5d96fd2d146ea1a156f9b8d"
server_key_iv: "cb7aea1c3c1eba126d9f2fad4e1b7305"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.374353 2096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 07:11:18.376806 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.378109 2096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:18.378237 2096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "d43efaf146fc44b6bf01f29a0e39911c"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "3c0f4d01a5d96fd2d146ea1a156f9b8d"
server_key_iv: "cb7aea1c3c1eba126d9f2fad4e1b7305"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.378410 2096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:18.402207 2096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:18.405351 2096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:18.405541 2096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:18.413020 2096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38137
I20260924 07:11:18.413039 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38137 every 8 connection(s)
I20260924 07:11:18.413993 2096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:18.416744 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:18.422348 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Bootstrap starting.
I20260924 07:11:18.424149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2096
I20260924 07:11:18.424252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:18.424497 392 external_mini_cluster.cc:1428] Setting key 1625672b8ff345f8fb6cc0303f45b1a7
I20260924 07:11:18.424955 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:18.425806 2174 log.cc:824] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:18.427887 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: No bootstrap required, opened a new log
I20260924 07:11:18.431897 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } }
I20260924 07:11:18.432217 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:18.432284 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d43efaf146fc44b6bf01f29a0e39911c, State: Initialized, Role: FOLLOWER
I20260924 07:11:18.432811 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } }
I20260924 07:11:18.432983 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:18.433056 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:18.433156 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:18.434343 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } }
I20260924 07:11:18.434685 2174 leader_election.cc:302] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d43efaf146fc44b6bf01f29a0e39911c; no voters:
I20260924 07:11:18.435002 2174 leader_election.cc:288] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:18.435104 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:18.435258 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [term 1 LEADER]: Becoming Leader. State: Replica: d43efaf146fc44b6bf01f29a0e39911c, State: Running, Role: LEADER
I20260924 07:11:18.435585 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } }
I20260924 07:11:18.436358 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:18.436796 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d43efaf146fc44b6bf01f29a0e39911c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } } }
I20260924 07:11:18.437129 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:18.437018 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d43efaf146fc44b6bf01f29a0e39911c. Latest consensus state: current_term: 1 leader_uuid: "d43efaf146fc44b6bf01f29a0e39911c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d43efaf146fc44b6bf01f29a0e39911c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38137 } } }
I20260924 07:11:18.437470 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:18.437767 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:18.439994 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.425672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.0.98.62:38137)
0924 07:11:18.426116 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:18.426133 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:18.426173 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:11:18.426964 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:18.426984 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:18.427335 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:18.427665 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:18.428503 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.429511 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.430756 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.431288 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.431479 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.434276 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:18.434282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:18.434297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:18.434328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:18.436186 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.436656 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.436662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.436669 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.436810 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.437130 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.437133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.437135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.437686 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:18.437892 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:18.438617 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 07:11:18.438946 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":111,"threads_started":1}
I20260924 07:11:18.442739 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:18.448993 2188 catalog_manager.cc:1347] Generated new cluster ID: e6a8144711b84166889cb4b6e55ca410
I20260924 07:11:18.449079 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:18.460543 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:18.461604 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:18.474393 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Generated new TSK 0
I20260924 07:11:18.475118 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:18.510105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38137
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:18.621872 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:18.622159 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:18.622207 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:18.626333 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:18.626402 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:18.626444 2200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:11:18.626461 2200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:18.626493 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:18.631817 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:18.633000 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:18.634006 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:18.643901 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:18.643895 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:18.643901 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:18.644347 2200 server_base.cc:1034] running on GCE node
I20260924 07:11:18.644793 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:18.645530 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:18.646708 2200 hybrid_clock.cc:648] HybridClock initialized: now 1790233878646680 us; error 40 us; skew 500 ppm
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:18.649616 2200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:18.650588 2200 webserver.cc:460] Webserver started at http://127.0.98.1:41901/ using document root <none> and password file <none>
I20260924 07:11:18.651149 2200 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:18.651207 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:18.651415 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:18.653528 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d20deaf68594a299050e8b8d26413b2"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "5152813278adfc503501339f682398a4"
server_key_iv: "40d368f59cf7ff2ad4b0e982c484e472"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.654050 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d20deaf68594a299050e8b8d26413b2"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "5152813278adfc503501339f682398a4"
server_key_iv: "40d368f59cf7ff2ad4b0e982c484e472"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.657831 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:18.660421 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.661777 2200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:18.661929 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "6d20deaf68594a299050e8b8d26413b2"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "5152813278adfc503501339f682398a4"
server_key_iv: "40d368f59cf7ff2ad4b0e982c484e472"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.662041 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:18.680454 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:18.683746 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:18.683967 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:18.685449 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:18.685501 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.685555 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:18.685585 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.695525 2200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43535
I20260924 07:11:18.695585 2330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43535 every 8 connection(s)
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:18.696511 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:18.700004 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.686326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44127 (local address 127.0.98.62:38137)
0924 07:11:18.686440 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:18.686443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:18.687627 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 07:11:18.688783 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:18.688788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:18.688851 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:18.688936 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:18.690789 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.691646 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.693454 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.693791 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.693918 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.696540 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:18.696543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:18.696545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:18.696571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:18.698034 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.698521 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.698524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.698525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.698599 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.698910 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.698913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.698914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.699128 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:18.699216 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:18.699749 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 07:11:18.699883 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:11:18.700793 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.686925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.1:44127)
0924 07:11:18.687486 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:18.687530 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:18.688504 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:18.689373 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:18.689380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:18.689860 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:18.690607 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.690628 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.691814 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.691818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.693278 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.693289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.693975 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.693980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.694152 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.694810 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:18.694826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:18.696368 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:18.698153 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.698158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.698170 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.698433 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.698663 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.698664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.698666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.698789 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.699246 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:18.699252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:18.699534 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 07:11:18.699697 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 07:11:18.699984 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":134,"threads_started":1}
I20260924 07:11:18.702049 2331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38137
I20260924 07:11:18.702204 2331 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:18.702745 2331 heartbeater.cc:511] Master 127.0.98.62:38137 requested a full tablet report, sending...
I20260924 07:11:18.704316 2128 ts_manager.cc:194] Registered new tserver with Master: 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535)
I20260924 07:11:18.705476 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44127
I20260924 07:11:18.706676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2200
I20260924 07:11:18.706784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:18.707077 392 external_mini_cluster.cc:1428] Setting key 7b78ab185287d67a1f2b19b54209b28e
I20260924 07:11:18.711598 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.704423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38571 (local address 127.0.98.62:38137)
0924 07:11:18.704537 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:18.704541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:18.704630 (+ 89us) server_negotiation.cc:407] Connection header received
0924 07:11:18.704732 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:18.704734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:18.704778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:18.704845 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:18.705249 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.705784 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.706726 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.707021 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.707122 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.708270 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:18.708274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:18.708277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:18.708302 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:18.709749 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.710292 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.710295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.710296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.710416 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.710752 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.710755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.710757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.710952 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:18.711040 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:18.711307 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:11:18.711447 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:18.711609 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.704287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.1:38571)
0924 07:11:18.704553 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:18.704566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:18.704663 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:18.704880 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:18.704883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:18.705063 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:18.705152 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.705157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.705900 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.705903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.706609 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.706619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.707182 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.707186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.707277 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.707833 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:18.707845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:18.708163 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:18.709884 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.709887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.709889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.710171 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.710541 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.710544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.710546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.710642 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.711051 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:18.711054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:18.711138 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 07:11:18.711238 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:11:18.711407 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":7,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:18.735129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38137
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:18.845975 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:18.846262 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:18.846313 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:18.850502 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:18.850569 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:18.850617 2340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:11:18.850638 2340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:18.850689 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:18.856171 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:18.857204 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:18.858191 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:18.867671 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:18.867700 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:18.867776 2340 server_base.cc:1034] running on GCE node
W20260924 07:11:18.867707 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:18.868300 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:18.868944 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:18.870107 2340 hybrid_clock.cc:648] HybridClock initialized: now 1790233878870075 us; error 41 us; skew 500 ppm
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:18.872887 2340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:18.873818 2340 webserver.cc:460] Webserver started at http://127.0.98.2:33053/ using document root <none> and password file <none>
I20260924 07:11:18.874366 2340 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:18.874455 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:18.874640 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:18.876720 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "e09f98bbe5f64fe5a9a10398e902404d"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "7612ae7a9be1a72907fa667fb4551902"
server_key_iv: "c0453706e1d3d7be84eb5db28da16bd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.877259 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e09f98bbe5f64fe5a9a10398e902404d"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "7612ae7a9be1a72907fa667fb4551902"
server_key_iv: "c0453706e1d3d7be84eb5db28da16bd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.880991 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:11:18.883487 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.884801 2340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:18.884912 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "e09f98bbe5f64fe5a9a10398e902404d"
format_stamp: "Formatted at 2026-09-24 07:11:18 on dist-test-slave-cnv3"
server_key: "7612ae7a9be1a72907fa667fb4551902"
server_key_iv: "c0453706e1d3d7be84eb5db28da16bd2"
server_key_version: "encryptionkey@0"
I20260924 07:11:18.885011 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:18.899940 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:18.903158 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:18.903323 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:18.904712 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:18.904760 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:18.904811 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:18.904840 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:18 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:18.914878 2340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33313
I20260924 07:11:18.914927 2470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33313 every 8 connection(s)
I20260924 07:11:18.915812 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:18.918648 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.905561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56397 (local address 127.0.98.62:38137)
0924 07:11:18.905686 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:18.905689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:18.906660 (+ 971us) server_negotiation.cc:407] Connection header received
0924 07:11:18.908007 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:18.908012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:18.908073 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:18.908172 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:18.909378 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.909896 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.911050 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.911363 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.911496 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.915029 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:18.915033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:18.915036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:18.915065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:18.916675 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.917146 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.917148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.917150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.917225 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.917543 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.917545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.917547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.917728 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:18.917807 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:18.918334 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 07:11:18.918519 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:11:18.919457 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.906047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.2:56397)
0924 07:11:18.906506 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:18.906559 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:18.907748 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:18.908337 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:18.908345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:18.908702 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:18.909238 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.909252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.910013 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.910016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.910930 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.910939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.911571 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.911576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.911788 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.913059 (+ 1271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:18.913081 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:18.914835 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:18.916783 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.916787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.916798 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.917061 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.917294 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.917296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.917298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.917421 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.917825 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:18.917831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:18.918091 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 07:11:18.918252 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 07:11:18.918506 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260924 07:11:18.920797 2471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38137
I20260924 07:11:18.920938 2471 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:18.921469 2471 heartbeater.cc:511] Master 127.0.98.62:38137 requested a full tablet report, sending...
I20260924 07:11:18.921510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2340
I20260924 07:11:18.921625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:18.921940 392 external_mini_cluster.cc:1428] Setting key 5c388450b1cb8d032dd04c559e7f3328
I20260924 07:11:18.922549 2127 ts_manager.cc:194] Registered new tserver with Master: e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313)
I20260924 07:11:18.923228 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56397
I20260924 07:11:18.930347 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.923262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53679 (local address 127.0.98.62:38137)
0924 07:11:18.923364 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:18.923368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:18.923652 (+ 284us) server_negotiation.cc:407] Connection header received
0924 07:11:18.923747 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:18.923750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:18.923791 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:18.923858 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:18.924336 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.925062 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.925900 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.926217 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.926305 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.927493 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:18.927496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:18.927498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:18.927525 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:18.928819 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.929265 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.929267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.929269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.929333 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:18.929594 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:18.929597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:18.929598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:18.929790 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:18.929863 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:18.930143 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 07:11:18.930233 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:11:18.930428 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:18.923236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.2:53679)
0924 07:11:18.923573 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:18.923586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:18.923676 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:18.923881 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:18.923884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:18.924106 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:18.924223 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.924229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.925203 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.925206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.925781 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:18.925790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:18.926317 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:18.926319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:18.926387 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:18.926873 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:18.926889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:18.927331 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:18.928933 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.928937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.928939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.929182 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.929414 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:18.929417 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:18.929419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:18.929513 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:18.929874 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:18.929877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:18.929987 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:11:18.930086 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:11:18.930222 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":179,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:18.951501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38137
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:19.061511 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:19.061793 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:19.061861 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:19.066195 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:19.066265 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:19.066335 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:11:19.066368 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:19.066422 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:19.071758 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:19.072760 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:19.073733 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:19.083207 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:19.083249 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:19.083210 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:19.083287 2480 server_base.cc:1034] running on GCE node
I20260924 07:11:19.084026 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:19.084645 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:19.085813 2480 hybrid_clock.cc:648] HybridClock initialized: now 1790233879085780 us; error 44 us; skew 500 ppm
Sep 24 07:11:19 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:19.088811 2480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:19.089772 2480 webserver.cc:460] Webserver started at http://127.0.98.3:44165/ using document root <none> and password file <none>
I20260924 07:11:19.090312 2480 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:19.090368 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:19.090549 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:19.092666 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2ef4c982a6c42518f347849121d8776"
format_stamp: "Formatted at 2026-09-24 07:11:19 on dist-test-slave-cnv3"
server_key: "aa841999e61ba350cf6605b742d053c7"
server_key_iv: "8b5601c0ed65e048d691982eb2004dad"
server_key_version: "encryptionkey@0"
I20260924 07:11:19.093200 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2ef4c982a6c42518f347849121d8776"
format_stamp: "Formatted at 2026-09-24 07:11:19 on dist-test-slave-cnv3"
server_key: "aa841999e61ba350cf6605b742d053c7"
server_key_iv: "8b5601c0ed65e048d691982eb2004dad"
server_key_version: "encryptionkey@0"
I20260924 07:11:19.096873 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:19.099337 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:19.100617 2480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:19.100716 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "d2ef4c982a6c42518f347849121d8776"
format_stamp: "Formatted at 2026-09-24 07:11:19 on dist-test-slave-cnv3"
server_key: "aa841999e61ba350cf6605b742d053c7"
server_key_iv: "8b5601c0ed65e048d691982eb2004dad"
server_key_version: "encryptionkey@0"
I20260924 07:11:19.100801 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:19.122960 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:19.126276 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:19.126449 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:19.127962 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:19.128016 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:19.128070 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:19.128098 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:19 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:19.137921 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39445
I20260924 07:11:19.137957 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39445 every 8 connection(s)
I20260924 07:11:19.138919 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:19.141350 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.128767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35871 (local address 127.0.98.62:38137)
0924 07:11:19.128886 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.128890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.129951 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 07:11:19.131004 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.131008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.131070 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.131132 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:19.132548 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.133133 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.134256 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.134557 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.134659 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.137918 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:19.137922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:19.137925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:19.137954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:19.139486 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.139989 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.139992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.139993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.140078 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.140322 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.140324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.140325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.140505 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:19.140581 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.141121 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.141230 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:11:19.142094 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.129321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.3:35871)
0924 07:11:19.129795 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.129839 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.130747 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.131556 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.131562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.131888 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:19.132406 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.132421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.133254 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.133258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.134137 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.134146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.134705 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.134710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.134901 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.136086 (+ 1185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:19.136102 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:19.137754 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:19.139587 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.139592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.139606 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.139905 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.140142 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.140143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.140145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.140262 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.140640 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:19.140645 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:19.140865 (+ 220us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.141032 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.141278 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260924 07:11:19.143270 2611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38137
I20260924 07:11:19.143437 2611 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:19.143978 2611 heartbeater.cc:511] Master 127.0.98.62:38137 requested a full tablet report, sending...
I20260924 07:11:19.145146 2128 ts_manager.cc:194] Registered new tserver with Master: d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445)
I20260924 07:11:19.145848 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35871
I20260924 07:11:19.147976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2480
I20260924 07:11:19.148087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:19.148336 392 external_mini_cluster.cc:1428] Setting key 80ae33b3cc31897ae54c2f9d68fa79ed
I20260924 07:11:19.150257 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:11:19.152967 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.145615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.98.3:43841)
0924 07:11:19.146192 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.146207 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.146298 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.146521 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.146523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.146689 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:19.146781 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.146786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.147779 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.147782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.148412 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.148423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.148891 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.148894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.148966 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.149476 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:19.149491 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:19.149926 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:19.151600 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.151603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.151605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.151842 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.152099 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.152101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.152102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.152173 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.152523 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:19.152525 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:19.152644 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.152719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.152824 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":79,"threads_started":1}
I20260924 07:11:19.153045 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.145663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43841 (local address 127.0.98.62:38137)
0924 07:11:19.145763 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.145766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.146267 (+ 501us) server_negotiation.cc:407] Connection header received
0924 07:11:19.146381 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.146384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.146428 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.146504 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:19.146881 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.147629 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.148537 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.148792 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.148876 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.150047 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:19.150051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:19.150053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:19.150079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:19.151505 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.151931 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.151935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.151937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.152022 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.152240 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.152242 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.152244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.152441 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:19.152523 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.152790 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.152900 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 07:11:19 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:11:19 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:19.174741 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.167112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.0.98.62:38137)
0924 07:11:19.167225 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.167229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.167239 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:11:19.167277 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.167279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.167315 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.167409 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:19.168092 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.168628 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.169428 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.169634 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.169719 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.172063 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:19.172066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:19.172068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:19.172092 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:19.173277 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.173647 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.173651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.173652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.173726 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.173975 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.173978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.173980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.174153 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:19.174271 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.174479 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.174583 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:11:19.177448 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:19.180608 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:19.194190 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.188495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.0.1:37148)
0924 07:11:19.188914 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.188965 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.189105 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.190271 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.190277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.190299 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.190481 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.190488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.191455 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.191458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.192663 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.192673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.193657 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.193661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.193761 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.193796 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.193907 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.194035 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":78,"thread_start_us":98,"threads_started":1}
I20260924 07:11:19.194199 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.188656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.0.98.3:39445)
0924 07:11:19.189378 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.189391 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.189413 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:11:19.189861 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.189867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.190084 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.190277 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.190606 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.191336 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.192798 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.193552 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.193649 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.193822 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.193942 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.194035 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":117,"threads_started":1}
I20260924 07:11:19.194608 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.189283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33313 (local address 127.0.0.1:46418)
0924 07:11:19.189564 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.189578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.189662 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.190343 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.190346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.190358 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.190481 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.190488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.192010 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.192014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.193217 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.193227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.194229 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.194232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.194328 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.194354 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.194402 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.194470 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":73,"thread_start_us":83,"threads_started":1}
I20260924 07:11:19.194751 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.188569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43535 (local address 127.0.0.1:48594)
0924 07:11:19.189073 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.189085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.189153 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.190427 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.190429 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.190445 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.190528 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.190533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.192025 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.192028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.193070 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.193079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.194418 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.194421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.194500 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.194519 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.194584 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.194643 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":5,"thread_start_us":78,"threads_started":1}
I20260924 07:11:19.195158 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.189550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48594 (local address 127.0.98.1:43535)
0924 07:11:19.189903 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.189918 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.189938 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:11:19.189999 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.190003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.190153 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.190276 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.190637 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.191903 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.193192 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.193985 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.194091 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.194854 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.194917 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.195017 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260924 07:11:19.195403 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.189437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.0.98.2:33313)
0924 07:11:19.189886 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.189902 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.189924 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:11:19.189999 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.190004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.190203 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.190634 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.190767 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.191890 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.193338 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.194105 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.194948 (+ 843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.195123 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.195172 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.195259 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":92,"threads_started":1}
I20260924 07:11:19.197148 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 8279a2e7312748a1826d519419a6828d (DEFAULT_TABLE table=test-table [id=0ffb2778df334bccba8af1c951cbdc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.198025 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8279a2e7312748a1826d519419a6828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.197687 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 8279a2e7312748a1826d519419a6828d (DEFAULT_TABLE table=test-table [id=0ffb2778df334bccba8af1c951cbdc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.198505 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8279a2e7312748a1826d519419a6828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.204689 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 8279a2e7312748a1826d519419a6828d (DEFAULT_TABLE table=test-table [id=0ffb2778df334bccba8af1c951cbdc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.205687 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8279a2e7312748a1826d519419a6828d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.205773 2635 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Bootstrap starting.
I20260924 07:11:19.206322 2634 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Bootstrap starting.
I20260924 07:11:19.208000 2635 tablet_bootstrap.cc:654] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.208251 2634 tablet_bootstrap.cc:654] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.208845 2635 log.cc:824] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:19.208933 2634 log.cc:824] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:19.210834 2635 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: No bootstrap required, opened a new log
I20260924 07:11:19.210834 2634 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: No bootstrap required, opened a new log
I20260924 07:11:19.211030 2635 ts_tablet_manager.cc:1397] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:11:19.211024 2634 ts_tablet_manager.cc:1397] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:11:19.211654 2638 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Bootstrap starting.
I20260924 07:11:19.214356 2638 tablet_bootstrap.cc:654] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.215078 2638 log.cc:824] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:19.214902 2635 raft_consensus.cc:348] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.214902 2634 raft_consensus.cc:348] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.215166 2635 raft_consensus.cc:374] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.215166 2634 raft_consensus.cc:374] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.215224 2635 raft_consensus.cc:729] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.215224 2634 raft_consensus.cc:729] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.215728 2635 consensus_queue.cc:260] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.215728 2634 consensus_queue.cc:260] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.216409 2471 heartbeater.cc:503] Master 127.0.98.62:38137 was elected leader, sending a full tablet report...
I20260924 07:11:19.216665 2611 heartbeater.cc:503] Master 127.0.98.62:38137 was elected leader, sending a full tablet report...
I20260924 07:11:19.216746 2634 ts_tablet_manager.cc:1428] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:11:19.216746 2635 ts_tablet_manager.cc:1428] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:11:19.218164 2638 tablet_bootstrap.cc:492] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: No bootstrap required, opened a new log
I20260924 07:11:19.218353 2638 ts_tablet_manager.cc:1397] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 07:11:19.222020 2638 raft_consensus.cc:348] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.222291 2638 raft_consensus.cc:374] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.222347 2638 raft_consensus.cc:729] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.222761 2638 consensus_queue.cc:260] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.223366 2331 heartbeater.cc:503] Master 127.0.98.62:38137 was elected leader, sending a full tablet report...
I20260924 07:11:19.223666 2638 ts_tablet_manager.cc:1428] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260924 07:11:19.389853 2612 tablet.cc:2369] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:19.416971 2472 tablet.cc:2369] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:19.448078 2332 tablet.cc:2369] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:19.514981 2640 raft_consensus.cc:482] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:19.515211 2640 raft_consensus.cc:504] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.516418 2640 leader_election.cc:288] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445), 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535)
I20260924 07:11:19.520287 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.516803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56307 (local address 127.0.98.1:43535)
0924 07:11:19.516933 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.516937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.517126 (+ 189us) server_negotiation.cc:407] Connection header received
0924 07:11:19.517245 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.517248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.517301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.517357 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.517696 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.518229 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519303 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.519706 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519783 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.519849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.520084 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.520164 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:19.520309 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.516689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.98.2:36521)
0924 07:11:19.517071 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.517084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.517194 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.517455 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.517458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.517491 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.517610 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.517615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.518308 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.518312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.519193 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.519202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.519849 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.519956 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.520012 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.520082 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.520155 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":157,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260924 07:11:19.520303 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.516825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36521 (local address 127.0.98.3:39445)
0924 07:11:19.516957 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.516961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.517174 (+ 213us) server_negotiation.cc:407] Connection header received
0924 07:11:19.517301 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.517303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.517358 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.517432 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.517716 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.518206 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519312 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.519739 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519836 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.519896 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.520107 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.520185 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:11:19.520327 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.516687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43535 (local address 127.0.98.2:56307)
0924 07:11:19.517026 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.517038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.517144 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.517405 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.517408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.517425 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.517585 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.517590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.518337 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.518341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.519184 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.519193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.519812 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.519816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.519907 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.519990 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.520046 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.520132 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":60,"spinlock_wait_cycles":24704,"thread_start_us":82,"threads_started":1}
I20260924 07:11:19.521080 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8279a2e7312748a1826d519419a6828d" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2" is_pre_election: true
I20260924 07:11:19.521090 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8279a2e7312748a1826d519419a6828d" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776" is_pre_election: true
I20260924 07:11:19.521384 2285 raft_consensus.cc:2454] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 0.
I20260924 07:11:19.521385 2565 raft_consensus.cc:2454] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 0.
I20260924 07:11:19.521871 2358 leader_election.cc:302] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:19.522148 2640 raft_consensus.cc:2781] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:19.522235 2640 raft_consensus.cc:482] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:19.522276 2640 raft_consensus.cc:3037] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.523273 2640 raft_consensus.cc:504] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.523646 2640 leader_election.cc:288] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 election: Requested vote from peers d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445), 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535)
I20260924 07:11:19.524037 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8279a2e7312748a1826d519419a6828d" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2"
I20260924 07:11:19.524037 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8279a2e7312748a1826d519419a6828d" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776"
I20260924 07:11:19.524173 2565 raft_consensus.cc:3037] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.524173 2285 raft_consensus.cc:3037] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.525521 2285 raft_consensus.cc:2454] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 1.
I20260924 07:11:19.525839 2565 raft_consensus.cc:2454] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 1.
I20260924 07:11:19.525863 2358 leader_election.cc:302] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:19.526023 2640 raft_consensus.cc:2781] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:19.526228 2640 raft_consensus.cc:686] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 1 LEADER]: Becoming Leader. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Running, Role: LEADER
I20260924 07:11:19.526537 2640 consensus_queue.cc:237] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } }
I20260924 07:11:19.529526 2127 catalog_manager.cc:5495] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d reported cstate change: term changed from 0 to 1, leader changed from <none> to e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e09f98bbe5f64fe5a9a10398e902404d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } health_report { overall_health: HEALTHY } } }
Sep 24 07:11:19 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:19.577000 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.569158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38137 (local address 127.0.0.1:58532)
0924 07:11:19.569425 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.569439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.569575 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.569788 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.569790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.570217 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:19.570313 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.570319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.571027 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.571031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.571636 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.571645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.572033 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.572036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.572108 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.572618 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:19.572631 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:19.574037 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:19.575529 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.575537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.575543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.575770 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.575995 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:19.575998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:19.575999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:19.576078 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:19.576457 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:19.576463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:19.576573 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.576683 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.576813 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260924 07:11:19.577003 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.569212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58532 (local address 127.0.98.62:38137)
0924 07:11:19.569327 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.569331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.569531 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:11:19.569656 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.569659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.569695 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.569760 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:19.570430 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.570928 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.571740 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.571939 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.572029 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.574180 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:19.574183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:19.574185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:19.574207 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:19.575430 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.575847 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.575849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.575851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.575925 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:19.576168 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:19.576170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:19.576172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:19.576345 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:19.576418 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.576725 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.576876 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":37}
I20260924 07:11:19.581007 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58532:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:11:19.581823 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:19.586730 2545 tablet_service.cc:1467] Processing CreateTablet for tablet cfc2548da7f94e71b66469c01d0a69d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b4a44e3e7ec4df2953fc7ca2fd363d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:19.586596 2265 tablet_service.cc:1467] Processing CreateTablet for tablet cfc2548da7f94e71b66469c01d0a69d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b4a44e3e7ec4df2953fc7ca2fd363d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:19.586670 2405 tablet_service.cc:1467] Processing CreateTablet for tablet cfc2548da7f94e71b66469c01d0a69d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0b4a44e3e7ec4df2953fc7ca2fd363d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:19.587337 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc2548da7f94e71b66469c01d0a69d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.587339 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc2548da7f94e71b66469c01d0a69d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.587339 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc2548da7f94e71b66469c01d0a69d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.589821 2638 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: Bootstrap starting.
I20260924 07:11:19.590071 2634 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: Bootstrap starting.
I20260924 07:11:19.590750 2638 tablet_bootstrap.cc:654] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.590873 2634 tablet_bootstrap.cc:654] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.591967 2635 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: Bootstrap starting.
I20260924 07:11:19.592103 2638 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: No bootstrap required, opened a new log
I20260924 07:11:19.592141 2634 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: No bootstrap required, opened a new log
I20260924 07:11:19.592195 2638 ts_tablet_manager.cc:1397] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:19.592216 2634 ts_tablet_manager.cc:1397] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:19.592638 2638 raft_consensus.cc:348] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.592658 2634 raft_consensus.cc:348] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.592746 2638 raft_consensus.cc:374] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.592767 2634 raft_consensus.cc:374] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.592775 2638 raft_consensus.cc:729] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.592806 2634 raft_consensus.cc:729] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.592893 2638 consensus_queue.cc:260] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.592926 2634 consensus_queue.cc:260] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.593158 2634 ts_tablet_manager.cc:1428] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:19.593159 2638 ts_tablet_manager.cc:1428] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:19.593197 2635 tablet_bootstrap.cc:654] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.593184 2641 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.593312 2641 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.593446 2642 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.593564 2642 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.594414 2635 tablet_bootstrap.cc:492] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: No bootstrap required, opened a new log
I20260924 07:11:19.594501 2635 ts_tablet_manager.cc:1397] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:19.594935 2635 raft_consensus.cc:348] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.595046 2635 raft_consensus.cc:374] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.595077 2635 raft_consensus.cc:729] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.595197 2635 consensus_queue.cc:260] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.595477 2635 ts_tablet_manager.cc:1428] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:19.595477 2640 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.595597 2640 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.700510 2641 raft_consensus.cc:482] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:19.700654 2641 raft_consensus.cc:504] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.701763 2641 leader_election.cc:288] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313)
I20260924 07:11:19.706166 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.702351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48327 (local address 127.0.98.2:33313)
0924 07:11:19.702592 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.702597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.702610 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:19.702661 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.702664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.702706 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.702788 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.703117 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.703908 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.704996 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.705594 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.705700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.705768 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.705955 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.706034 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260924 07:11:19.706187 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.702121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39385 (local address 127.0.98.1:43535)
0924 07:11:19.702257 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.702261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.702412 (+ 151us) server_negotiation.cc:407] Connection header received
0924 07:11:19.702574 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.702578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.702619 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.702678 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:19.702911 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.703675 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.705196 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.705744 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.705832 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.705872 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.706016 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.706088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:11:19.706164 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.701978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43535 (local address 127.0.98.3:39385)
0924 07:11:19.702314 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.702329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.702441 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.702700 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.702703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.702718 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.702816 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.702821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.703816 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.703819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.705083 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.705093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.705843 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.705846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.705922 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.705940 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.705999 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.706048 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260924 07:11:19.706169 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.702004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33313 (local address 127.0.98.3:48327)
0924 07:11:19.702392 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.702404 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.702463 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.702887 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.702890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.702906 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:19.703003 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.703008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.704012 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.704016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.704852 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.704861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.705738 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.705740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.705825 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.705852 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.705930 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.706006 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":68,"threads_started":1}
I20260924 07:11:19.706769 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc2548da7f94e71b66469c01d0a69d4" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2" is_pre_election: true
I20260924 07:11:19.706780 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc2548da7f94e71b66469c01d0a69d4" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d" is_pre_election: true
I20260924 07:11:19.706907 2285 raft_consensus.cc:2454] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 0.
I20260924 07:11:19.706975 2425 raft_consensus.cc:2454] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 0.
I20260924 07:11:19.707242 2498 leader_election.cc:302] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, d2ef4c982a6c42518f347849121d8776; no voters:
I20260924 07:11:19.707582 2641 raft_consensus.cc:2781] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:19.707654 2641 raft_consensus.cc:482] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:19.707691 2641 raft_consensus.cc:3037] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.708628 2641 raft_consensus.cc:504] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.708962 2641 leader_election.cc:288] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 election: Requested vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313)
I20260924 07:11:19.709334 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc2548da7f94e71b66469c01d0a69d4" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d"
I20260924 07:11:19.709462 2425 raft_consensus.cc:3037] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.709499 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc2548da7f94e71b66469c01d0a69d4" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2"
I20260924 07:11:19.709638 2285 raft_consensus.cc:3037] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.710553 2285 raft_consensus.cc:2454] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 1.
I20260924 07:11:19.710637 2425 raft_consensus.cc:2454] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 1.
I20260924 07:11:19.710821 2498 leader_election.cc:302] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, d2ef4c982a6c42518f347849121d8776; no voters:
I20260924 07:11:19.710973 2641 raft_consensus.cc:2781] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:19.711150 2641 raft_consensus.cc:686] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [term 1 LEADER]: Becoming Leader. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Running, Role: LEADER
I20260924 07:11:19.711529 2641 consensus_queue.cc:237] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.713438 2659 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: TxnStatusTablet state changed. Reason: New leader d2ef4c982a6c42518f347849121d8776. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.713528 2659 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:19.713719 2661 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:11:19.713907 2661 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:11:19.714754 2127 catalog_manager.cc:5495] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2ef4c982a6c42518f347849121d8776 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } health_report { overall_health: HEALTHY } } }
I20260924 07:11:19.754251 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.750178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.98.3:39445)
0924 07:11:19.750317 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.750321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.750416 (+ 95us) server_negotiation.cc:407] Connection header received
0924 07:11:19.750621 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.750623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.750712 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.750790 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:19.751017 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.751519 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.752721 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.752933 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.753008 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.753347 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.753877 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.753974 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.754052 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.754110 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:19.754295 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.750053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.0.1:37158)
0924 07:11:19.750352 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.750368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.750487 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.750821 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.750824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.750832 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:19.750937 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.750943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.751611 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.751614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.752574 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.752585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.753030 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.753034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.753123 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.753176 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.753966 (+ 790us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.753979 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.754045 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.754131 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260924 07:11:19.759639 2285 raft_consensus.cc:1261] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Refusing update from remote peer d2ef4c982a6c42518f347849121d8776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.759769 2425 raft_consensus.cc:1261] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Refusing update from remote peer d2ef4c982a6c42518f347849121d8776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.760380 2659 consensus_queue.cc:1035] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:19.760577 2641 consensus_queue.cc:1035] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:19.763998 2640 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: TxnStatusTablet state changed. Reason: New leader d2ef4c982a6c42518f347849121d8776. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.764129 2640 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.764540 2642 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: TxnStatusTablet state changed. Reason: New leader d2ef4c982a6c42518f347849121d8776. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.764652 2642 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.767434 2641 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.767567 2641 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:19.768594 2659 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.768697 2659 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P d2ef4c982a6c42518f347849121d8776: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:19.770071 2642 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.770180 2642 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P 6d20deaf68594a299050e8b8d26413b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.770303 2665 mvcc.cc:205] Tried to move back new op lower bound from 7332797971487576064 to 7332797971301224448. Current Snapshot: MvccSnapshot[applied={T|T < 7332797971487576064}]
I20260924 07:11:19.771757 2640 tablet_replica.cc:439] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } } }
I20260924 07:11:19.771872 2640 tablet_replica.cc:442] T cfc2548da7f94e71b66469c01d0a69d4 P e09f98bbe5f64fe5a9a10398e902404d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:19.772375 2664 mvcc.cc:205] Tried to move back new op lower bound from 7332797971487576064 to 7332797971301224448. Current Snapshot: MvccSnapshot[applied={T|T < 7332797971487576064 or (T in {7332797971487576064})}]
I20260924 07:11:19.772539 2666 mvcc.cc:205] Tried to move back new op lower bound from 7332797971487576064 to 7332797971301224448. Current Snapshot: MvccSnapshot[applied={T|T < 7332797971487576064}]
I20260924 07:11:19.776861 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.772977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.0.98.2:33313)
0924 07:11:19.773078 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.773083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.773189 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:11:19.773352 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.773356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.773413 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.773479 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:19.773847 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.774571 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.775565 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.775790 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.775874 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.776122 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.776514 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.776609 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.776687 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.776740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:11:19.788425 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.784855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39053 (local address 127.0.98.3:39445)
0924 07:11:19.784963 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.784967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.785155 (+ 188us) server_negotiation.cc:407] Connection header received
0924 07:11:19.785250 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.785253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.785307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.785383 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:19.785584 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.786369 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.787247 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.787618 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.787743 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.788000 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.788093 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.788161 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.788229 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.788282 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:11:19.788450 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.784700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.98.2:39053)
0924 07:11:19.785069 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.785080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.785176 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.785387 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.785389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.785394 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:19.785503 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.785507 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.786488 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.786490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.787138 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.787146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.787728 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.787731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.787816 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.787862 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.788164 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.788168 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.788219 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.788315 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260924 07:11:19.796083 2565 raft_consensus.cc:1261] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Refusing update from remote peer e09f98bbe5f64fe5a9a10398e902404d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.796561 2285 raft_consensus.cc:1261] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Refusing update from remote peer e09f98bbe5f64fe5a9a10398e902404d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.796620 2640 consensus_queue.cc:1035] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:19.797008 2645 consensus_queue.cc:1035] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:19.821431 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.816854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52999 (local address 127.0.98.2:33313)
0924 07:11:19.816967 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:19.816971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:19.817124 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:11:19.817251 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:19.817254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:19.817305 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:19.817383 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:19.817746 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.818507 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.819810 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.820030 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.820124 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.820511 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.820706 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.820802 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:19.821208 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 07:11:19.821271 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:11:19.821460 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:19.816756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33313 (local address 127.0.98.3:52999)
0924 07:11:19.817051 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:19.817063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:19.817150 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:19.817436 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:19.817439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:19.817447 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:19.817619 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.817635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.818640 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.818644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.819670 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:19.819682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:19.820155 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:19.820159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:19.820275 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:19.820330 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:19.820975 (+ 645us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:19.820984 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:11:19.821185 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:11:19.821281 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260924 07:11:19.850330 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:19.850534 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:19.853529 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Sending DeleteTablet for 3 replicas of tablet 8279a2e7312748a1826d519419a6828d
I20260924 07:11:19.854550 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 8279a2e7312748a1826d519419a6828d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260924 07:11:19.854547 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8279a2e7312748a1826d519419a6828d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46418
I20260924 07:11:19.855252 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 8279a2e7312748a1826d519419a6828d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260924 07:11:19.855310 2686 tablet_replica.cc:330] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: stopping tablet replica
I20260924 07:11:19.855759 2686 raft_consensus.cc:2229] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:19.856381 2685 tablet_replica.cc:330] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: stopping tablet replica
I20260924 07:11:19.856513 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:11:19.856663 2685 raft_consensus.cc:2229] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:19.856817 2685 raft_consensus.cc:2258] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:11:19.856957 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:19.858137 2685 ts_tablet_manager.cc:1905] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:19.861240 2685 ts_tablet_manager.cc:1918] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:19.861340 2685 log.cc:1196] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/8279a2e7312748a1826d519419a6828d
I20260924 07:11:19.861616 2685 ts_tablet_manager.cc:1939] T 8279a2e7312748a1826d519419a6828d P d2ef4c982a6c42518f347849121d8776: Deleting consensus metadata
I20260924 07:11:19.862960 2112 catalog_manager.cc:4841] TS d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445): tablet 8279a2e7312748a1826d519419a6828d (table test-table [id=0ffb2778df334bccba8af1c951cbdc44]) successfully deleted
I20260924 07:11:19.863328 2686 raft_consensus.cc:2258] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:19.863818 2405 tablet_service.cc:1467] Processing CreateTablet for tablet a68782e93d824488b2eced0532cef451 (DEFAULT_TABLE table=test-table [id=d735b0a6bc454f5080b0fe7b19469267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.864084 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68782e93d824488b2eced0532cef451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.866480 2635 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Bootstrap starting.
I20260924 07:11:19.866704 2265 tablet_service.cc:1467] Processing CreateTablet for tablet a68782e93d824488b2eced0532cef451 (DEFAULT_TABLE table=test-table [id=d735b0a6bc454f5080b0fe7b19469267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.866948 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68782e93d824488b2eced0532cef451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.867492 2635 tablet_bootstrap.cc:654] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.868891 2635 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: No bootstrap required, opened a new log
I20260924 07:11:19.868978 2635 ts_tablet_manager.cc:1397] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 07:11:19.869405 2635 raft_consensus.cc:348] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.869509 2635 raft_consensus.cc:374] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.869529 2638 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Bootstrap starting.
I20260924 07:11:19.869536 2635 raft_consensus.cc:729] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.869653 2635 consensus_queue.cc:260] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.869889 2635 ts_tablet_manager.cc:1428] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:11:19.870429 2638 tablet_bootstrap.cc:654] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.871311 2545 tablet_service.cc:1467] Processing CreateTablet for tablet a68782e93d824488b2eced0532cef451 (DEFAULT_TABLE table=test-table [id=d735b0a6bc454f5080b0fe7b19469267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:19.871570 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68782e93d824488b2eced0532cef451. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:19.873749 2634 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Bootstrap starting.
I20260924 07:11:19.874518 2634 tablet_bootstrap.cc:654] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:19.875391 2686 ts_tablet_manager.cc:1905] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:19.876093 2634 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: No bootstrap required, opened a new log
I20260924 07:11:19.876173 2634 ts_tablet_manager.cc:1397] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:19.876597 2634 raft_consensus.cc:348] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.876695 2634 raft_consensus.cc:374] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.876721 2634 raft_consensus.cc:729] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.876829 2634 consensus_queue.cc:260] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.877060 2634 ts_tablet_manager.cc:1428] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:19.878278 2686 ts_tablet_manager.cc:1918] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:19.878371 2686 log.cc:1196] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/8279a2e7312748a1826d519419a6828d
I20260924 07:11:19.878646 2686 ts_tablet_manager.cc:1939] T 8279a2e7312748a1826d519419a6828d P e09f98bbe5f64fe5a9a10398e902404d: Deleting consensus metadata
I20260924 07:11:19.879581 2687 tablet_replica.cc:330] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: stopping tablet replica
I20260924 07:11:19.879910 2687 raft_consensus.cc:2229] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:19.880129 2687 raft_consensus.cc:2258] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:19.880807 2112 catalog_manager.cc:4841] TS e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313): tablet 8279a2e7312748a1826d519419a6828d (table test-table [id=0ffb2778df334bccba8af1c951cbdc44]) successfully deleted
I20260924 07:11:19.882089 2687 ts_tablet_manager.cc:1905] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:19.884111 2638 tablet_bootstrap.cc:492] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: No bootstrap required, opened a new log
I20260924 07:11:19.884217 2638 ts_tablet_manager.cc:1397] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260924 07:11:19.884665 2638 raft_consensus.cc:348] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.884786 2638 raft_consensus.cc:374] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:19.884815 2638 raft_consensus.cc:729] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Initialized, Role: FOLLOWER
I20260924 07:11:19.884932 2638 consensus_queue.cc:260] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.885190 2638 ts_tablet_manager.cc:1428] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:19.887143 2687 ts_tablet_manager.cc:1918] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:19.887248 2687 log.cc:1196] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/8279a2e7312748a1826d519419a6828d
I20260924 07:11:19.887564 2687 ts_tablet_manager.cc:1939] T 8279a2e7312748a1826d519419a6828d P 6d20deaf68594a299050e8b8d26413b2: Deleting consensus metadata
I20260924 07:11:19.888787 2114 catalog_manager.cc:4841] TS 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535): tablet 8279a2e7312748a1826d519419a6828d (table test-table [id=0ffb2778df334bccba8af1c951cbdc44]) successfully deleted
I20260924 07:11:19.933755 2640 raft_consensus.cc:482] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:19.933889 2640 raft_consensus.cc:504] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.934314 2640 leader_election.cc:288] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445)
I20260924 07:11:19.934799 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68782e93d824488b2eced0532cef451" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776" is_pre_election: true
I20260924 07:11:19.934816 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68782e93d824488b2eced0532cef451" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2" is_pre_election: true
I20260924 07:11:19.934950 2565 raft_consensus.cc:2454] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 0.
I20260924 07:11:19.934950 2284 raft_consensus.cc:2454] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 0.
I20260924 07:11:19.935298 2356 leader_election.cc:302] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ef4c982a6c42518f347849121d8776, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:19.935498 2640 raft_consensus.cc:2781] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:19.935549 2640 raft_consensus.cc:482] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:19.935586 2640 raft_consensus.cc:3037] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.936609 2640 raft_consensus.cc:504] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.936996 2640 leader_election.cc:288] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 election: Requested vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445)
I20260924 07:11:19.937427 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68782e93d824488b2eced0532cef451" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2"
I20260924 07:11:19.937543 2284 raft_consensus.cc:3037] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.938525 2284 raft_consensus.cc:2454] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 1.
I20260924 07:11:19.938783 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68782e93d824488b2eced0532cef451" candidate_uuid: "e09f98bbe5f64fe5a9a10398e902404d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776"
I20260924 07:11:19.938820 2358 leader_election.cc:302] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:19.938911 2565 raft_consensus.cc:3037] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:19.939002 2640 raft_consensus.cc:2781] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:19.939074 2640 raft_consensus.cc:686] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 1 LEADER]: Becoming Leader. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Running, Role: LEADER
I20260924 07:11:19.939210 2640 consensus_queue.cc:237] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:19.940030 2565 raft_consensus.cc:2454] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09f98bbe5f64fe5a9a10398e902404d in term 1.
I20260924 07:11:19.941015 2127 catalog_manager.cc:5495] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d reported cstate change: term changed from 0 to 1, leader changed from <none> to e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e09f98bbe5f64fe5a9a10398e902404d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:19.972935 2565 raft_consensus.cc:1261] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Refusing update from remote peer e09f98bbe5f64fe5a9a10398e902404d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.972980 2284 raft_consensus.cc:1261] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Refusing update from remote peer e09f98bbe5f64fe5a9a10398e902404d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:19.973286 2640 consensus_queue.cc:1035] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:19.973554 2680 consensus_queue.cc:1035] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:11:19.987068 2666 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:11:21.541462 2127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:21.541675 2127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:21.544615 2127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Sending DeleteTablet for 3 replicas of tablet a68782e93d824488b2eced0532cef451
I20260924 07:11:21.545317 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet a68782e93d824488b2eced0532cef451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46418
I20260924 07:11:21.545320 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet a68782e93d824488b2eced0532cef451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260924 07:11:21.545506 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet a68782e93d824488b2eced0532cef451 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260924 07:11:21.545814 2718 tablet_replica.cc:330] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: stopping tablet replica
I20260924 07:11:21.545895 2719 tablet_replica.cc:330] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: stopping tablet replica
I20260924 07:11:21.545975 2718 raft_consensus.cc:2229] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:21.546032 2719 raft_consensus.cc:2229] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:21.546167 2719 raft_consensus.cc:2258] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:21.546219 2718 raft_consensus.cc:2258] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:21.546531 2717 tablet_replica.cc:330] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: stopping tablet replica
I20260924 07:11:21.546670 2717 raft_consensus.cc:2229] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:21.546797 2717 raft_consensus.cc:2258] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:21.547451 2718 ts_tablet_manager.cc:1905] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.547494 2717 ts_tablet_manager.cc:1905] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.547489 2719 ts_tablet_manager.cc:1905] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.550678 2719 ts_tablet_manager.cc:1918] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:11:21.550678 2718 ts_tablet_manager.cc:1918] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:11:21.550776 2718 log.cc:1196] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/a68782e93d824488b2eced0532cef451
I20260924 07:11:21.550776 2719 log.cc:1196] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/a68782e93d824488b2eced0532cef451
I20260924 07:11:21.550983 2717 ts_tablet_manager.cc:1918] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:11:21.551034 2719 ts_tablet_manager.cc:1939] T a68782e93d824488b2eced0532cef451 P d2ef4c982a6c42518f347849121d8776: Deleting consensus metadata
I20260924 07:11:21.551059 2718 ts_tablet_manager.cc:1939] T a68782e93d824488b2eced0532cef451 P e09f98bbe5f64fe5a9a10398e902404d: Deleting consensus metadata
I20260924 07:11:21.551060 2717 log.cc:1196] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/a68782e93d824488b2eced0532cef451
I20260924 07:11:21.551285 2717 ts_tablet_manager.cc:1939] T a68782e93d824488b2eced0532cef451 P 6d20deaf68594a299050e8b8d26413b2: Deleting consensus metadata
I20260924 07:11:21.552179 2114 catalog_manager.cc:4841] TS 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535): tablet a68782e93d824488b2eced0532cef451 (table test-table [id=d735b0a6bc454f5080b0fe7b19469267]) successfully deleted
I20260924 07:11:21.552179 2112 catalog_manager.cc:4841] TS d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445): tablet a68782e93d824488b2eced0532cef451 (table test-table [id=d735b0a6bc454f5080b0fe7b19469267]) successfully deleted
I20260924 07:11:21.552524 2112 catalog_manager.cc:4841] TS e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313): tablet a68782e93d824488b2eced0532cef451 (table test-table [id=d735b0a6bc454f5080b0fe7b19469267]) successfully deleted
Sep 24 07:11:21 dist-test-slave-cnv3 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:11:21 dist-test-slave-cnv3 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233881, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:21.579169 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.569500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.98.62:38137)
0924 07:11:21.569834 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.569844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.569863 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:11:21.570036 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.570040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.570138 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.570233 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:21.571035 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.571788 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.572678 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.572921 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.573013 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.575767 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:21.575770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:21.575773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:21.575810 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:21.577468 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:21.577890 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:21.577892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:21.577894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:21.577971 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:21.578298 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:21.578300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:21.578302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:21.578510 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:21.578620 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.578858 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.578976 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260924 07:11:21.582448 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:21.583034 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:21.588289 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 (DEFAULT_TABLE table=test-table [id=b0cf538a8c06495193a6daca38a63747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.588366 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 (DEFAULT_TABLE table=test-table [id=b0cf538a8c06495193a6daca38a63747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.588387 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 (DEFAULT_TABLE table=test-table [id=b0cf538a8c06495193a6daca38a63747]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.588673 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d1adf2c76d4db98b99db68c971ed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.588673 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d1adf2c76d4db98b99db68c971ed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.588701 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51d1adf2c76d4db98b99db68c971ed36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.591054 2728 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Bootstrap starting.
I20260924 07:11:21.591784 2730 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Bootstrap starting.
I20260924 07:11:21.592164 2728 tablet_bootstrap.cc:654] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.592658 2730 tablet_bootstrap.cc:654] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.593343 2728 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: No bootstrap required, opened a new log
I20260924 07:11:21.593425 2728 ts_tablet_manager.cc:1397] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:21.593592 2729 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Bootstrap starting.
I20260924 07:11:21.593899 2728 raft_consensus.cc:348] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.594036 2728 raft_consensus.cc:374] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.594069 2728 raft_consensus.cc:729] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.594251 2728 consensus_queue.cc:260] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.594496 2730 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: No bootstrap required, opened a new log
I20260924 07:11:21.594588 2729 tablet_bootstrap.cc:654] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.594686 2730 ts_tablet_manager.cc:1397] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:21.595171 2728 ts_tablet_manager.cc:1428] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:21.595230 2730 raft_consensus.cc:348] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.595357 2730 raft_consensus.cc:374] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.595422 2730 raft_consensus.cc:729] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.595558 2730 consensus_queue.cc:260] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.595955 2730 ts_tablet_manager.cc:1428] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:21.596849 2729 tablet_bootstrap.cc:492] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: No bootstrap required, opened a new log
I20260924 07:11:21.596943 2729 ts_tablet_manager.cc:1397] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:21.597406 2729 raft_consensus.cc:348] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.597527 2729 raft_consensus.cc:374] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.597559 2729 raft_consensus.cc:729] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.597683 2729 consensus_queue.cc:260] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.597976 2729 ts_tablet_manager.cc:1428] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:21.742785 2694 raft_consensus.cc:482] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:21.742978 2694 raft_consensus.cc:504] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.743535 2694 leader_election.cc:288] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313)
I20260924 07:11:21.744175 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d1adf2c76d4db98b99db68c971ed36" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2" is_pre_election: true
I20260924 07:11:21.744191 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d1adf2c76d4db98b99db68c971ed36" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d" is_pre_election: true
I20260924 07:11:21.744371 2425 raft_consensus.cc:2454] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 0.
I20260924 07:11:21.744391 2284 raft_consensus.cc:2454] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 0.
I20260924 07:11:21.744765 2496 leader_election.cc:302] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, d2ef4c982a6c42518f347849121d8776, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:21.744972 2694 raft_consensus.cc:2781] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:21.745034 2694 raft_consensus.cc:482] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:21.745061 2694 raft_consensus.cc:3037] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.746312 2694 raft_consensus.cc:504] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.746701 2694 leader_election.cc:288] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 election: Requested vote from peers 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535), e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313)
I20260924 07:11:21.747102 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d1adf2c76d4db98b99db68c971ed36" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d"
I20260924 07:11:21.747190 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51d1adf2c76d4db98b99db68c971ed36" candidate_uuid: "d2ef4c982a6c42518f347849121d8776" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d20deaf68594a299050e8b8d26413b2"
I20260924 07:11:21.747259 2425 raft_consensus.cc:3037] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.747327 2284 raft_consensus.cc:3037] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.748596 2284 raft_consensus.cc:2454] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 1.
I20260924 07:11:21.748596 2425 raft_consensus.cc:2454] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2ef4c982a6c42518f347849121d8776 in term 1.
I20260924 07:11:21.748903 2496 leader_election.cc:302] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2ef4c982a6c42518f347849121d8776, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:21.749070 2694 raft_consensus.cc:2781] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:21.749154 2694 raft_consensus.cc:686] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 1 LEADER]: Becoming Leader. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Running, Role: LEADER
I20260924 07:11:21.749305 2694 consensus_queue.cc:237] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.751256 2128 catalog_manager.cc:5495] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2ef4c982a6c42518f347849121d8776 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d2ef4c982a6c42518f347849121d8776" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } health_report { overall_health: HEALTHY } } }
I20260924 07:11:21.799252 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.795673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.0.98.3:39445)
0924 07:11:21.795972 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.795980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.795993 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:21.796038 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.796040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.796087 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.796177 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:21.796432 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.797036 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.797979 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.798240 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.798325 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.798700 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.798805 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.798871 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.798985 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.799049 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":131,"threads_started":1}
I20260924 07:11:21.810170 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.807037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34077 (local address 127.0.98.3:39445)
0924 07:11:21.807155 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.807160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.807409 (+ 249us) server_negotiation.cc:407] Connection header received
0924 07:11:21.807552 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.807554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.807600 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.807668 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:21.807867 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.808408 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.809253 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.809473 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.809543 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.809772 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.809851 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.809896 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.809966 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.810015 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":37}
I20260924 07:11:21.810177 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.806985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.98.3:34077)
0924 07:11:21.807305 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:21.807318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:21.807467 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:21.807680 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:21.807682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:21.807688 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:21.807790 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.807795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.808509 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.808512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.809162 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.809171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.809558 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.809561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.809656 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.809683 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.809909 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.809913 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:21.809959 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:11:21.810037 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":155,"threads_started":1}
I20260924 07:11:21.816989 2425 raft_consensus.cc:1261] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Refusing update from remote peer d2ef4c982a6c42518f347849121d8776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:21.816989 2284 raft_consensus.cc:1261] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Refusing update from remote peer d2ef4c982a6c42518f347849121d8776: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:21.817431 2703 consensus_queue.cc:1035] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:21.817616 2698 consensus_queue.cc:1035] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:21.858264 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:21.858515 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:21.861291 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Sending DeleteTablet for 3 replicas of tablet 51d1adf2c76d4db98b99db68c971ed36
I20260924 07:11:21.862089 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260924 07:11:21.862116 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260924 07:11:21.862337 2719 tablet_replica.cc:330] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: stopping tablet replica
I20260924 07:11:21.862479 2719 raft_consensus.cc:2229] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:21.862886 2719 raft_consensus.cc:2258] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:21.863001 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:11:21.863139 2717 tablet_replica.cc:330] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: stopping tablet replica
I20260924 07:11:21.863281 2717 raft_consensus.cc:2229] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:21.863458 2717 raft_consensus.cc:2258] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:11:21.863595 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:21.864327 2719 ts_tablet_manager.cc:1905] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.867473 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 51d1adf2c76d4db98b99db68c971ed36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46418
I20260924 07:11:21.867699 2718 tablet_replica.cc:330] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: stopping tablet replica
I20260924 07:11:21.867844 2718 raft_consensus.cc:2229] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:21.867995 2718 raft_consensus.cc:2258] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:21.868623 2719 ts_tablet_manager.cc:1918] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:21.868714 2719 log.cc:1196] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/51d1adf2c76d4db98b99db68c971ed36
I20260924 07:11:21.868974 2719 ts_tablet_manager.cc:1939] T 51d1adf2c76d4db98b99db68c971ed36 P d2ef4c982a6c42518f347849121d8776: Deleting consensus metadata
I20260924 07:11:21.869575 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 (DEFAULT_TABLE table=test-table [id=5088faba20c74a348d2911dec555bdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.869855 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5b44c6502b42abb10a74f7e7e541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.869876 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 (DEFAULT_TABLE table=test-table [id=5088faba20c74a348d2911dec555bdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.870174 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5b44c6502b42abb10a74f7e7e541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.872031 2728 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Bootstrap starting.
I20260924 07:11:21.872812 2730 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: Bootstrap starting.
I20260924 07:11:21.872834 2728 tablet_bootstrap.cc:654] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.873854 2730 tablet_bootstrap.cc:654] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.873956 2728 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: No bootstrap required, opened a new log
I20260924 07:11:21.874049 2728 ts_tablet_manager.cc:1397] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:21.874509 2728 raft_consensus.cc:348] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.874644 2728 raft_consensus.cc:374] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.874679 2728 raft_consensus.cc:729] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ef4c982a6c42518f347849121d8776, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.874815 2728 consensus_queue.cc:260] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.875095 2728 ts_tablet_manager.cc:1428] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:21.875321 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 (DEFAULT_TABLE table=test-table [id=5088faba20c74a348d2911dec555bdbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:21.875682 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d5b44c6502b42abb10a74f7e7e541c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:21.876049 2112 catalog_manager.cc:4841] TS d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445): tablet 51d1adf2c76d4db98b99db68c971ed36 (table test-table [id=b0cf538a8c06495193a6daca38a63747]) successfully deleted
I20260924 07:11:21.876909 2717 ts_tablet_manager.cc:1905] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.878441 2729 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: Bootstrap starting.
I20260924 07:11:21.879551 2729 tablet_bootstrap.cc:654] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:21.880204 2718 ts_tablet_manager.cc:1905] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:21.880630 2730 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: No bootstrap required, opened a new log
I20260924 07:11:21.880750 2730 ts_tablet_manager.cc:1397] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 07:11:21.881294 2730 raft_consensus.cc:348] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.881426 2730 raft_consensus.cc:374] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.881461 2730 raft_consensus.cc:729] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.881615 2730 consensus_queue.cc:260] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.882004 2730 ts_tablet_manager.cc:1428] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 07:11:21.882611 2718 ts_tablet_manager.cc:1918] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:21.882692 2718 log.cc:1196] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/51d1adf2c76d4db98b99db68c971ed36
I20260924 07:11:21.882931 2718 ts_tablet_manager.cc:1939] T 51d1adf2c76d4db98b99db68c971ed36 P e09f98bbe5f64fe5a9a10398e902404d: Deleting consensus metadata
I20260924 07:11:21.883729 2112 catalog_manager.cc:4841] TS e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313): tablet 51d1adf2c76d4db98b99db68c971ed36 (table test-table [id=b0cf538a8c06495193a6daca38a63747]) successfully deleted
I20260924 07:11:21.884191 2717 ts_tablet_manager.cc:1918] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:21.884256 2717 log.cc:1196] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/51d1adf2c76d4db98b99db68c971ed36
I20260924 07:11:21.884426 2717 ts_tablet_manager.cc:1939] T 51d1adf2c76d4db98b99db68c971ed36 P 6d20deaf68594a299050e8b8d26413b2: Deleting consensus metadata
I20260924 07:11:21.885135 2114 catalog_manager.cc:4841] TS 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535): tablet 51d1adf2c76d4db98b99db68c971ed36 (table test-table [id=b0cf538a8c06495193a6daca38a63747]) successfully deleted
I20260924 07:11:21.888355 2729 tablet_bootstrap.cc:492] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: No bootstrap required, opened a new log
I20260924 07:11:21.888468 2729 ts_tablet_manager.cc:1397] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260924 07:11:21.888942 2729 raft_consensus.cc:348] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.889060 2729 raft_consensus.cc:374] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:21.889082 2729 raft_consensus.cc:729] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f98bbe5f64fe5a9a10398e902404d, State: Initialized, Role: FOLLOWER
I20260924 07:11:21.889228 2729 consensus_queue.cc:260] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.889515 2729 ts_tablet_manager.cc:1428] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:21.923516 2734 raft_consensus.cc:482] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:21.923718 2734 raft_consensus.cc:504] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.925370 2734 leader_election.cc:288] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313), d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445)
I20260924 07:11:21.931758 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.925790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54475 (local address 127.0.98.2:33313)
0924 07:11:21.926074 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.926092 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.926365 (+ 273us) server_negotiation.cc:407] Connection header received
0924 07:11:21.926602 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.926606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.926664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.926725 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:21.927152 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.928094 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.930063 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.930779 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.930857 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.930932 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.931439 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.931572 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260924 07:11:21.931831 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.926272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47223 (local address 127.0.98.3:39445)
0924 07:11:21.926407 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.926411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.926617 (+ 206us) server_negotiation.cc:407] Connection header received
0924 07:11:21.926807 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.926810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.926863 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.926908 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:21.927316 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.928124 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.930068 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.930749 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.930835 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.930926 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.931441 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.931681 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:11:21.932430 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.926133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.98.1:47223)
0924 07:11:21.926523 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:21.926536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:21.926694 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:21.927040 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:21.927043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:21.927058 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:21.927203 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.927209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.928293 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.928296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.929893 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.929909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.931056 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.931060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.931250 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.931299 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:11:21.932133 (+ 834us) client_negotiation.cc:240] Negotiation successful
0924 07:11:21.932237 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":54,"thread_start_us":107,"threads_started":1}
I20260924 07:11:21.933096 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.925691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33313 (local address 127.0.98.1:54475)
0924 07:11:21.926263 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:21.926279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:21.926460 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:21.926883 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:21.926887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:21.926904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:21.927048 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.927054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.928250 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.928254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.929896 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.929909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.931056 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.931060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.931234 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.931294 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 07:11:21.932420 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0924 07:11:21.932948 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":198,"thread_start_us":130,"threads_started":1}
I20260924 07:11:21.933157 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5b44c6502b42abb10a74f7e7e541c6" candidate_uuid: "6d20deaf68594a299050e8b8d26413b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776" is_pre_election: true
I20260924 07:11:21.933346 2565 raft_consensus.cc:2454] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20deaf68594a299050e8b8d26413b2 in term 0.
I20260924 07:11:21.934536 2216 leader_election.cc:302] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, d2ef4c982a6c42518f347849121d8776; no voters:
I20260924 07:11:21.934932 2734 raft_consensus.cc:2781] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:21.935004 2734 raft_consensus.cc:482] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:21.935043 2734 raft_consensus.cc:3037] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.935263 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5b44c6502b42abb10a74f7e7e541c6" candidate_uuid: "6d20deaf68594a299050e8b8d26413b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d" is_pre_election: true
I20260924 07:11:21.935447 2425 raft_consensus.cc:2454] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d20deaf68594a299050e8b8d26413b2 in term 0.
I20260924 07:11:21.936713 2734 raft_consensus.cc:504] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.937130 2734 leader_election.cc:288] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [CANDIDATE]: Term 1 election: Requested vote from peers e09f98bbe5f64fe5a9a10398e902404d (127.0.98.2:33313), d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445)
I20260924 07:11:21.937541 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5b44c6502b42abb10a74f7e7e541c6" candidate_uuid: "6d20deaf68594a299050e8b8d26413b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e09f98bbe5f64fe5a9a10398e902404d"
I20260924 07:11:21.937686 2425 raft_consensus.cc:3037] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.937661 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d5b44c6502b42abb10a74f7e7e541c6" candidate_uuid: "6d20deaf68594a299050e8b8d26413b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ef4c982a6c42518f347849121d8776"
I20260924 07:11:21.937806 2565 raft_consensus.cc:3037] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:21.938896 2425 raft_consensus.cc:2454] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20deaf68594a299050e8b8d26413b2 in term 1.
I20260924 07:11:21.939199 2216 leader_election.cc:302] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d20deaf68594a299050e8b8d26413b2, e09f98bbe5f64fe5a9a10398e902404d; no voters:
I20260924 07:11:21.939412 2734 raft_consensus.cc:2781] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:21.939555 2565 raft_consensus.cc:2454] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d20deaf68594a299050e8b8d26413b2 in term 1.
I20260924 07:11:21.939651 2734 raft_consensus.cc:686] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [term 1 LEADER]: Becoming Leader. State: Replica: 6d20deaf68594a299050e8b8d26413b2, State: Running, Role: LEADER
I20260924 07:11:21.940011 2734 consensus_queue.cc:237] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } }
I20260924 07:11:21.943111 2128 catalog_manager.cc:5495] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6d20deaf68594a299050e8b8d26413b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d20deaf68594a299050e8b8d26413b2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:21.968328 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.963485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47854 (local address 127.0.98.1:43535)
0924 07:11:21.963957 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.963964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.963974 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:11:21.964062 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.964064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.964134 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.964265 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:21.964554 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.965147 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.966368 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.966635 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.966715 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.967034 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.967813 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.967915 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.968029 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.968097 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":154,"threads_started":1}
I20260924 07:11:21.981567 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.977779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39445 (local address 127.0.98.1:38657)
0924 07:11:21.978198 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:21.978214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:21.978352 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:21.978603 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:21.978606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:21.978615 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:21.978794 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.978801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.979644 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.979648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.980296 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:21.980305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.980751 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.980754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:21.980851 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.980905 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.981214 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.981220 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:21.981276 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:11:21.981376 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":159,"threads_started":1}
I20260924 07:11:21.981556 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:21.978012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38657 (local address 127.0.98.3:39445)
0924 07:11:21.978168 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:21.978210 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:21.978294 (+ 84us) server_negotiation.cc:407] Connection header received
0924 07:11:21.978459 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:21.978462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:21.978515 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:21.978580 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:21.978908 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.979533 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.980410 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:21.980631 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:21.980721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:21.981046 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:21.981137 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:21.981201 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:21.981295 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 07:11:21.981376 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 07:11:21.989252 2425 raft_consensus.cc:1261] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Refusing update from remote peer 6d20deaf68594a299050e8b8d26413b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:21.989252 2565 raft_consensus.cc:1261] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Refusing update from remote peer 6d20deaf68594a299050e8b8d26413b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:21.989879 2750 consensus_queue.cc:1035] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ef4c982a6c42518f347849121d8776" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:21.990129 2734 consensus_queue.cc:1035] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e09f98bbe5f64fe5a9a10398e902404d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:22.011078 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:22.007308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39183 (local address 127.0.98.1:43535)
0924 07:11:22.007467 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:22.007472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:22.007495 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:11:22.007544 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:22.007547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:22.007600 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:22.007952 (+ 352us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:22.008054 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:22.008892 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:22.009785 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:22.010123 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:22.010509 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:22.010579 (+ 70us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:22.010670 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:22.010728 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:22.010849 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:11:22.010917 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:11:22.011078 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:22.007100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43535 (local address 127.0.98.3:39183)
0924 07:11:22.007237 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:22.007254 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:22.007344 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:22.007713 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:22.007717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:22.007726 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:22.007863 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:22.007869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:22.009017 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:22.009020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:22.009663 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:22.009674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:22.010248 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:22.010252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:22.010343 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:22.010392 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:22.010744 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:22.010751 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:11:22.010820 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:11:22.010929 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:11:22.011967 2700 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:11:23.582036 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:23.582203 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:23.584812 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d43efaf146fc44b6bf01f29a0e39911c: Sending DeleteTablet for 3 replicas of tablet 4d5b44c6502b42abb10a74f7e7e541c6
I20260924 07:11:23.585420 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46418
I20260924 07:11:23.585454 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37148
I20260924 07:11:23.585454 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d5b44c6502b42abb10a74f7e7e541c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48594
I20260924 07:11:23.585777 2784 tablet_replica.cc:330] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: stopping tablet replica
I20260924 07:11:23.585776 2783 tablet_replica.cc:330] T 4d5b44c6502b42abb10a74f7e7e541c6 P 6d20deaf68594a299050e8b8d26413b2: stopping tablet replica
I20260924 07:11:23.585819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2200
I20260924 07:11:23.586735 2784 raft_consensus.cc:2229] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:23.586874 2784 raft_consensus.cc:2258] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:23.587533 2784 ts_tablet_manager.cc:1905] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:23.589933 2784 ts_tablet_manager.cc:1918] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:11:23.589998 2784 log.cc:1196] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/4d5b44c6502b42abb10a74f7e7e541c6
I20260924 07:11:23.590200 2784 ts_tablet_manager.cc:1939] T 4d5b44c6502b42abb10a74f7e7e541c6 P d2ef4c982a6c42518f347849121d8776: Deleting consensus metadata
I20260924 07:11:23.590878 2785 tablet_replica.cc:330] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: stopping tablet replica
I20260924 07:11:23.591087 2785 raft_consensus.cc:2229] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:23.591081 2112 catalog_manager.cc:4841] TS d2ef4c982a6c42518f347849121d8776 (127.0.98.3:39445): tablet 4d5b44c6502b42abb10a74f7e7e541c6 (table test-table [id=5088faba20c74a348d2911dec555bdbe]) successfully deleted
I20260924 07:11:23.591224 2785 raft_consensus.cc:2258] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:23.591831 2785 ts_tablet_manager.cc:1905] T 4d5b44c6502b42abb10a74f7e7e541c6 P e09f98bbe5f64fe5a9a10398e902404d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:11:23.593071 2114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43535 (error 108)
I20260924 07:11:23.593849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2340
I20260924 07:11:23.594133 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:23.593729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43535 (local address 127.0.0.1:47860)
0924 07:11:23.593969 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:23.594040 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
W20260924 07:11:23.594938 2114 catalog_manager.cc:4584] TS 6d20deaf68594a299050e8b8d26413b2 (127.0.98.1:43535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d5b44c6502b42abb10a74f7e7e541c6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43535: connect: Connection refused (error 111)
I20260924 07:11:23.599462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2480
I20260924 07:11:23.600849 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:23.599484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33313 (local address 127.0.0.1:44934)
0924 07:11:23.600718 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:23.600774 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:11:23.605244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2096
2026-09-24T07:11:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6473 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:23 dist-test-slave-cnv3 krb5kdc[2790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:23 dist-test-slave-cnv3 krb5kdc[2790](info): set up 2 sockets
Sep 24 07:11:23 dist-test-slave-cnv3 krb5kdc[2790](info): commencing operation
krb5kdc: starting...
W20260924 07:11:24.571429 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:24 dist-test-slave-cnv3 krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:24.694952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:24.805651 2803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:24.805946 2803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:24.805986 2803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:24.810041 2803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:24.810109 2803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:24.810127 2803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:24.810143 2803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:24.810158 2803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:24.815675 2803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:24.816743 2803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:24.817806 2803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:24.826838 2811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:24.826783 2809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:24.827045 2803 server_base.cc:1034] running on GCE node
W20260924 07:11:24.826916 2808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:24.827749 2803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:24.828943 2803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:24.830191 2803 hybrid_clock.cc:648] HybridClock initialized: now 1790233884830101 us; error 35 us; skew 500 ppm
Sep 24 07:11:24 dist-test-slave-cnv3 krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:24.833194 2803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:24.834180 2803 webserver.cc:460] Webserver started at http://127.0.98.62:38443/ using document root <none> and password file <none>
I20260924 07:11:24.834776 2803 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:24.834842 2803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:24.835067 2803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:24.837361 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "cf392af609034cd9a5d0f1d8948d7833"
format_stamp: "Formatted at 2026-09-24 07:11:24 on dist-test-slave-cnv3"
server_key: "2acc2226f5e98b00e4b44a9bdabb15b3"
server_key_iv: "20b824c8216ec5f67a7c5672250c3f19"
server_key_version: "encryptionkey@0"
I20260924 07:11:24.837927 2803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf392af609034cd9a5d0f1d8948d7833"
format_stamp: "Formatted at 2026-09-24 07:11:24 on dist-test-slave-cnv3"
server_key: "2acc2226f5e98b00e4b44a9bdabb15b3"
server_key_iv: "20b824c8216ec5f67a7c5672250c3f19"
server_key_version: "encryptionkey@0"
I20260924 07:11:24.842015 2803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:11:24.844841 2818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:24.846359 2803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:24.846494 2803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "cf392af609034cd9a5d0f1d8948d7833"
format_stamp: "Formatted at 2026-09-24 07:11:24 on dist-test-slave-cnv3"
server_key: "2acc2226f5e98b00e4b44a9bdabb15b3"
server_key_iv: "20b824c8216ec5f67a7c5672250c3f19"
server_key_version: "encryptionkey@0"
I20260924 07:11:24.846606 2803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:24.864025 2803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:24.867234 2803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:24.867472 2803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:24.874426 2803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38899
I20260924 07:11:24.874444 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38899 every 8 connection(s)
I20260924 07:11:24.875522 2803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:24.878472 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:24.881378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2803
I20260924 07:11:24.881479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:24.881709 392 external_mini_cluster.cc:1428] Setting key 00e6080cdfc3a12ace9e60b1f0913f99
I20260924 07:11:24.884915 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833: Bootstrap starting.
I20260924 07:11:24.887938 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:24.888837 2871 log.cc:824] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:24.891356 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833: No bootstrap required, opened a new log
Sep 24 07:11:24 dist-test-slave-cnv3 krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:24.895530 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } }
I20260924 07:11:24.895829 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:24.895892 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf392af609034cd9a5d0f1d8948d7833, State: Initialized, Role: FOLLOWER
I20260924 07:11:24.896481 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } }
I20260924 07:11:24.896648 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:24.896718 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:24.896849 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:24.898020 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } }
I20260924 07:11:24.898409 2871 leader_election.cc:302] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf392af609034cd9a5d0f1d8948d7833; no voters:
I20260924 07:11:24.898533 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:24.883142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.98.62:38899)
0924 07:11:24.883635 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:24.883656 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:24.883701 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:11:24.884525 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:24.884550 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:24.885032 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:24.885472 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:24.886199 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:24.887771 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:24.888828 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:24.889525 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:24.889672 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:24.892498 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:24.892505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:24.892525 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:24.892560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:24.895142 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:24.895691 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:24.895697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:24.895704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:24.895861 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:24.896146 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:24.896149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:24.896151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:24.896685 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:24.896913 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:24.897178 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 07:11:24.897492 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":142,"threads_started":1}
I20260924 07:11:24.898748 2871 leader_election.cc:288] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:24.898907 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:24.899122 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [term 1 LEADER]: Becoming Leader. State: Replica: cf392af609034cd9a5d0f1d8948d7833, State: Running, Role: LEADER
I20260924 07:11:24.899523 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } }
I20260924 07:11:24.900094 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:24.900628 2876 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf392af609034cd9a5d0f1d8948d7833. Latest consensus state: current_term: 1 leader_uuid: "cf392af609034cd9a5d0f1d8948d7833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } } }
I20260924 07:11:24.900837 2876 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:24.901278 2884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:24.901500 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf392af609034cd9a5d0f1d8948d7833" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf392af609034cd9a5d0f1d8948d7833" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38899 } } }
I20260924 07:11:24.901682 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:24.904588 2884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:24.910502 2884 catalog_manager.cc:1347] Generated new cluster ID: 8a471cc4cd464be7b01a62647745888c
I20260924 07:11:24.910586 2884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:24.939462 2884 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:24.940567 2884 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:24.945344 2884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf392af609034cd9a5d0f1d8948d7833: Generated new TSK 0
I20260924 07:11:24.946069 2884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:24.986044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:25.120630 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:25.120936 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:25.120981 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:25.125191 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:25.125269 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:25.125330 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:25.130726 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:25.131909 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:25.133013 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:25.143092 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:25.143203 2897 server_base.cc:1034] running on GCE node
W20260924 07:11:25.143092 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:25.143095 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:25.144006 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:25.144758 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:25.145932 2897 hybrid_clock.cc:648] HybridClock initialized: now 1790233885145912 us; error 38 us; skew 500 ppm
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:25.148753 2897 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:25.149672 2897 webserver.cc:460] Webserver started at http://127.0.98.1:39687/ using document root <none> and password file <none>
I20260924 07:11:25.150218 2897 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:25.150277 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:25.150444 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:25.152598 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "9121418bca624106bd4ae30c95a19221"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "17b9408b052988ad849e1a8e6e7f020f"
server_key_iv: "e07edddb51a1ef3f3c55bd634e108af4"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.153136 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9121418bca624106bd4ae30c95a19221"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "17b9408b052988ad849e1a8e6e7f020f"
server_key_iv: "e07edddb51a1ef3f3c55bd634e108af4"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.156896 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:25.159407 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.160650 2897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:25.160754 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "9121418bca624106bd4ae30c95a19221"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "17b9408b052988ad849e1a8e6e7f020f"
server_key_iv: "e07edddb51a1ef3f3c55bd634e108af4"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.160844 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:25.174793 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:25.177733 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:25.177886 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:25.178450 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:25.179237 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:25.179282 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.179319 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:25.179333 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.188030 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44997
I20260924 07:11:25.188074 3025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44997 every 8 connection(s)
I20260924 07:11:25.188931 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:25.193218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 2897
I20260924 07:11:25.193400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:25.193675 392 external_mini_cluster.cc:1428] Setting key 3d936aa12f03a287aeb430a444552825
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:25.203715 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.190751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58761 (local address 127.0.98.62:38899)
0924 07:11:25.190892 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:25.190895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:25.191765 (+ 870us) server_negotiation.cc:407] Connection header received
0924 07:11:25.193312 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:25.193315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:25.193370 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:25.193468 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:25.194979 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.195626 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.197184 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.197431 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.197522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.200042 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:25.200045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:25.200047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:25.200072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:25.201524 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.202066 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.202070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.202071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.202146 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.202492 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.202496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.202498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.202744 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:25.202845 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:25.203419 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 07:11:25.203586 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:11:25.204592 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.191154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38899 (local address 127.0.98.1:58761)
0924 07:11:25.191610 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:25.191655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:25.192968 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:25.193755 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:25.193778 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:25.194233 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:25.194835 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.194852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.195773 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.195777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.197062 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.197072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.197585 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.197589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.197753 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.198319 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:25.198336 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:25.199876 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:25.201686 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.201692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.201706 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.201965 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.202237 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.202240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.202242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.202381 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.202845 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:25.202850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:25.203100 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:11:25.203268 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 07:11:25.203709 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:25.205842 3026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38899
I20260924 07:11:25.206107 3026 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:25.206686 3026 heartbeater.cc:511] Master 127.0.98.62:38899 requested a full tablet report, sending...
I20260924 07:11:25.208367 2835 ts_manager.cc:194] Registered new tserver with Master: 9121418bca624106bd4ae30c95a19221 (127.0.98.1:44997)
I20260924 07:11:25.209517 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:25.223152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:25.333590 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:25.333907 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:25.333966 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:25.338238 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:25.338327 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:25.338408 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:25.343911 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:25.345038 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:25.346076 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:25.356003 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:25.356000 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:25.356020 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:25.356383 3031 server_base.cc:1034] running on GCE node
I20260924 07:11:25.356881 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:25.357582 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:25.358780 3031 hybrid_clock.cc:648] HybridClock initialized: now 1790233885358752 us; error 47 us; skew 500 ppm
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:25.361747 3031 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:25.362735 3031 webserver.cc:460] Webserver started at http://127.0.98.2:42955/ using document root <none> and password file <none>
I20260924 07:11:25.363299 3031 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:25.363365 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:25.363587 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:25.365760 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "baaf9cba00d948738636546452f54bd1"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "b7c3da7929091f4c932a7b207d7226bf"
server_key_iv: "fc77eed831fde7befe3f958dbc9909ba"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.366353 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "baaf9cba00d948738636546452f54bd1"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "b7c3da7929091f4c932a7b207d7226bf"
server_key_iv: "fc77eed831fde7befe3f958dbc9909ba"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.370378 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 07:11:25.372967 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.374405 3031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:25.374531 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "baaf9cba00d948738636546452f54bd1"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "b7c3da7929091f4c932a7b207d7226bf"
server_key_iv: "fc77eed831fde7befe3f958dbc9909ba"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.374660 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:25.392767 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:25.395877 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:25.396051 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:25.396644 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:25.397697 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:25.397778 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.397869 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:25.397891 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.407042 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33055
I20260924 07:11:25.407104 3159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33055 every 8 connection(s)
I20260924 07:11:25.408203 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:25.409897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3031
I20260924 07:11:25.410007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:25.410286 392 external_mini_cluster.cc:1428] Setting key 9de9f05303233566b900510a57580c95
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:25.423486 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.410585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33431 (local address 127.0.98.62:38899)
0924 07:11:25.410724 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:25.410728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:25.411462 (+ 734us) server_negotiation.cc:407] Connection header received
0924 07:11:25.413185 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:25.413188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:25.413297 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:25.413394 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:25.414645 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.415450 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.416661 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.416997 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.417110 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.419786 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:25.419789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:25.419791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:25.419815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:25.421199 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.421717 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.421720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.421721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.421819 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.422194 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.422197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.422198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.422453 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:25.422547 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:25.423155 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 07:11:25.423279 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:11:25.424489 3162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.410822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38899 (local address 127.0.98.2:33431)
0924 07:11:25.411305 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:25.411351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:25.412931 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:25.413615 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:25.413624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:25.413990 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:25.414498 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.414512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.415570 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.415573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.416529 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.416539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.417163 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.417166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.417332 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.417939 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:25.417955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:25.419617 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:25.421317 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.421324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.421340 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.421641 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.421925 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.421928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.421930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.422087 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.422573 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:25.422578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:25.422878 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 07:11:25.423102 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 07:11:25.423519 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":137,"threads_started":1}
I20260924 07:11:25.425884 3160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38899
I20260924 07:11:25.426112 3160 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:25.426690 3160 heartbeater.cc:511] Master 127.0.98.62:38899 requested a full tablet report, sending...
I20260924 07:11:25.427915 2835 ts_manager.cc:194] Registered new tserver with Master: baaf9cba00d948738636546452f54bd1 (127.0.98.2:33055)
I20260924 07:11:25.428701 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:25.439668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:25.549095 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:25.549392 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:25.549453 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:25.553824 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:25.553900 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:25.553984 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:25.559517 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:25.560664 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:25.561730 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:25.571798 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:25.571856 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:25.571798 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:25.572036 3165 server_base.cc:1034] running on GCE node
I20260924 07:11:25.572598 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:25.573277 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:25.574474 3165 hybrid_clock.cc:648] HybridClock initialized: now 1790233885574449 us; error 38 us; skew 500 ppm
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:25.577401 3165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:25.578496 3165 webserver.cc:460] Webserver started at http://127.0.98.3:39821/ using document root <none> and password file <none>
I20260924 07:11:25.579103 3165 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:25.579174 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:25.579423 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:25.581552 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "701595e346994a3bb241ce3941446784"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "8b9cf970118e47beb55d72c6b2430b00"
server_key_iv: "b4cd41e61160acaac3214ffbfb65142b"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.582108 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "701595e346994a3bb241ce3941446784"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "8b9cf970118e47beb55d72c6b2430b00"
server_key_iv: "b4cd41e61160acaac3214ffbfb65142b"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.585968 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:11:25.588590 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.590013 3165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:25.590137 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "701595e346994a3bb241ce3941446784"
format_stamp: "Formatted at 2026-09-24 07:11:25 on dist-test-slave-cnv3"
server_key: "8b9cf970118e47beb55d72c6b2430b00"
server_key_iv: "b4cd41e61160acaac3214ffbfb65142b"
server_key_version: "encryptionkey@0"
I20260924 07:11:25.590246 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:25.607658 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:25.611016 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:25.611220 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:25.611869 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:25.612746 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:25.612800 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.612864 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:25.612892 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:25.621536 3165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39669
I20260924 07:11:25.621613 3293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39669 every 8 connection(s)
I20260924 07:11:25.622460 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:25.626083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3165
I20260924 07:11:25.626187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:25.626526 392 external_mini_cluster.cc:1428] Setting key a1b6d35a3ba46d949f7758ec9869212a
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[2790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:25.637033 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.624507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37657 (local address 127.0.98.62:38899)
0924 07:11:25.624656 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:25.624659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:25.625575 (+ 916us) server_negotiation.cc:407] Connection header received
0924 07:11:25.627069 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:25.627072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:25.627120 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:25.627192 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:25.628577 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.629119 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.630288 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.630503 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.630593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.633298 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:25.633359 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:25.633361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:25.633388 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:25.634868 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.635439 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.635442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.635443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.635521 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:25.635856 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:25.635859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:25.635861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:25.636069 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:25.636156 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:25.636735 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 07:11:25.636866 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:11:25.637938 3296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:25.624895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38899 (local address 127.0.98.3:37657)
0924 07:11:25.625401 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:25.625446 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:25.626850 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:25.627445 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:25.627455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:25.627893 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:25.628442 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.628462 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.629249 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.629252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.630164 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:25.630173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:25.630650 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:25.630653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:25.630822 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:25.631527 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:25.631545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:25.633111 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:25.635008 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.635014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.635033 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.635300 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.635611 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:25.635614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:25.635616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:25.635758 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:25.636173 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:25.636180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:25.636447 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:11:25.636674 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 07:11:25.637057 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":118,"threads_started":1}
I20260924 07:11:25.639418 3294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38899
I20260924 07:11:25.639708 3294 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:25.640282 3294 heartbeater.cc:511] Master 127.0.98.62:38899 requested a full tablet report, sending...
I20260924 07:11:25.641374 2835 ts_manager.cc:194] Registered new tserver with Master: 701595e346994a3bb241ce3941446784 (127.0.98.3:39669)
I20260924 07:11:25.641932 2835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37657
I20260924 07:11:25.652684 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:11:25.665546 2874 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:25.664042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.62:38899)
0924 07:11:25.664171 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:25.664174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:25.664303 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:11:25.664382 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:25.664384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:25.664429 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:25.664509 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:25.665430 (+ 921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37658: BlockingRecv error: recv got EOF from 127.0.0.1:37658 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:25.666536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2897
I20260924 07:11:25.671528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3031
I20260924 07:11:25.676234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3165
I20260924 07:11:25.680624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 2803
2026-09-24T07:11:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2074 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[3306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[3306](info): set up 2 sockets
Sep 24 07:11:25 dist-test-slave-cnv3 krb5kdc[3306](info): commencing operation
krb5kdc: starting...
W20260924 07:11:26.643555 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:26 dist-test-slave-cnv3 krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:26.768247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:26.879909 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:26.880187 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:26.880228 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:26.884251 3319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:26.884308 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:26.884327 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:26.884342 3319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:26.884356 3319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:26.889573 3319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:26.890601 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:26.891649 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:26.900005 3324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:26.900024 3325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:26.900024 3327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:26.900066 3319 server_base.cc:1034] running on GCE node
I20260924 07:11:26.900866 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:26.902001 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:26.903164 3319 hybrid_clock.cc:648] HybridClock initialized: now 1790233886903148 us; error 30 us; skew 500 ppm
Sep 24 07:11:26 dist-test-slave-cnv3 krb5kdc[3306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:26.905799 3319 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:26.906634 3319 webserver.cc:460] Webserver started at http://127.0.98.62:46183/ using document root <none> and password file <none>
I20260924 07:11:26.907160 3319 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:26.907223 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:26.907451 3319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:26.909519 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "daa4960aafe6426094af4a0cbc6e87b5"
format_stamp: "Formatted at 2026-09-24 07:11:26 on dist-test-slave-cnv3"
server_key: "57d794426ed584b3136a874a6f693ecf"
server_key_iv: "5511a9ad14c730271180593349945140"
server_key_version: "encryptionkey@0"
I20260924 07:11:26.910040 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "daa4960aafe6426094af4a0cbc6e87b5"
format_stamp: "Formatted at 2026-09-24 07:11:26 on dist-test-slave-cnv3"
server_key: "57d794426ed584b3136a874a6f693ecf"
server_key_iv: "5511a9ad14c730271180593349945140"
server_key_version: "encryptionkey@0"
I20260924 07:11:26.913619 3319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:11:26.916100 3334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:26.917402 3319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:26.917590 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "daa4960aafe6426094af4a0cbc6e87b5"
format_stamp: "Formatted at 2026-09-24 07:11:26 on dist-test-slave-cnv3"
server_key: "57d794426ed584b3136a874a6f693ecf"
server_key_iv: "5511a9ad14c730271180593349945140"
server_key_version: "encryptionkey@0"
I20260924 07:11:26.917695 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:26.933270 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:26.936820 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:26.937000 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:26.944108 3319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43659
I20260924 07:11:26.944394 3386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43659 every 8 connection(s)
I20260924 07:11:26.945127 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:26.948026 3387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:26.953626 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5: Bootstrap starting.
I20260924 07:11:26.954419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3319
I20260924 07:11:26.954514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:26.954728 392 external_mini_cluster.cc:1428] Setting key 7dfdbe6844ffae993940ad60454314e5
I20260924 07:11:26.956403 3387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:26.957283 3387 log.cc:824] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:26.959474 3387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5: No bootstrap required, opened a new log
I20260924 07:11:26.963445 3387 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } }
I20260924 07:11:26.963676 3387 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:26.963712 3387 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa4960aafe6426094af4a0cbc6e87b5, State: Initialized, Role: FOLLOWER
I20260924 07:11:26.964147 3387 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } }
Sep 24 07:11:26 dist-test-slave-cnv3 krb5kdc[3306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:26.964259 3387 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:26.964298 3387 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:26.964378 3387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:26.965405 3387 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } }
I20260924 07:11:26.965777 3387 leader_election.cc:302] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa4960aafe6426094af4a0cbc6e87b5; no voters:
I20260924 07:11:26.966122 3387 leader_election.cc:288] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:26.966229 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:26.966391 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [term 1 LEADER]: Becoming Leader. State: Replica: daa4960aafe6426094af4a0cbc6e87b5, State: Running, Role: LEADER
I20260924 07:11:26.966683 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } }
I20260924 07:11:26.967495 3387 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:26.968237 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa4960aafe6426094af4a0cbc6e87b5. Latest consensus state: current_term: 1 leader_uuid: "daa4960aafe6426094af4a0cbc6e87b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } } }
I20260924 07:11:26.968355 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:26.968708 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa4960aafe6426094af4a0cbc6e87b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa4960aafe6426094af4a0cbc6e87b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43659 } } }
I20260924 07:11:26.968765 3401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:26.968791 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:26.970633 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:26.955922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.98.62:43659)
0924 07:11:26.956412 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:26.956431 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:26.956478 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:11:26.957310 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:26.957334 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:26.957780 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:26.958192 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:26.958858 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:26.960326 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:26.961332 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:26.962102 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:26.962279 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:26.964840 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:26.964845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:26.964860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:26.964892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:26.966766 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:26.967235 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:26.967242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:26.967248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:26.967364 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:26.967701 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:26.967704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:26.967706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:26.968266 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:26.969159 (+ 893us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:26.969321 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:11:26.969653 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":109,"threads_started":1}
I20260924 07:11:26.972123 3401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:26.973697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:11:26.978145 3401 catalog_manager.cc:1347] Generated new cluster ID: 89c6064f1a7042d2a82d8426cdd082d6
I20260924 07:11:26.978215 3401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:26.991097 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:26.986441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.62:43659)
0924 07:11:26.986582 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:26.986587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:26.986792 (+ 205us) server_negotiation.cc:407] Connection header received
0924 07:11:26.986914 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:26.986918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:26.986962 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:26.987036 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:26.987595 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:26.988154 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:26.988988 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:26.989222 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:26.989311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:26.990120 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:26.990123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:26.990126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:26.990628 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:26.990659 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:26.990675 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:26.990758 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:26.990866 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:11:26.990934 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:11:26.996433 3401 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:26.997730 3401 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:27.009421 3401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa4960aafe6426094af4a0cbc6e87b5: Generated new TSK 0
I20260924 07:11:27.010327 3401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:11:27.024631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3319
2026-09-24T07:11:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1342 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:27 dist-test-slave-cnv3 krb5kdc[3420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:27 dist-test-slave-cnv3 krb5kdc[3420](info): set up 2 sockets
Sep 24 07:11:27 dist-test-slave-cnv3 krb5kdc[3420](info): commencing operation
krb5kdc: starting...
W20260924 07:11:27.983184 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:28.119077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:28.231302 3433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:28.231637 3433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:28.231704 3433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:28.235884 3433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:28.235958 3433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:28.235983 3433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:28.236006 3433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:28.236027 3433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:28.242177 3433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:28.243458 3433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:28.244580 3433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:28.253495 3439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.253484 3441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.253458 3438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.253777 3433 server_base.cc:1034] running on GCE node
I20260924 07:11:28.254501 3433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:28.255780 3433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:28.256976 3433 hybrid_clock.cc:648] HybridClock initialized: now 1790233888256960 us; error 33 us; skew 500 ppm
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:28.259946 3433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:28.260866 3433 webserver.cc:460] Webserver started at http://127.0.98.62:35821/ using document root <none> and password file <none>
I20260924 07:11:28.261407 3433 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:28.261467 3433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:28.261651 3433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:28.263792 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "b90e4ac6bb2f492da758955126798295"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "eba5553547533df2c3901810c2a82327"
server_key_iv: "780b92109b8ec16825e49b7cc185ef04"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.264351 3433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "b90e4ac6bb2f492da758955126798295"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "eba5553547533df2c3901810c2a82327"
server_key_iv: "780b92109b8ec16825e49b7cc185ef04"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.268474 3433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:28.271275 3448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.272846 3433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:28.272961 3433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "b90e4ac6bb2f492da758955126798295"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "eba5553547533df2c3901810c2a82327"
server_key_iv: "780b92109b8ec16825e49b7cc185ef04"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.273061 3433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:28.287238 3433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:28.290597 3433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:28.290812 3433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:28.297832 3433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34723
I20260924 07:11:28.298152 3500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34723 every 8 connection(s)
I20260924 07:11:28.298838 3433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:28.301775 3501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:28.305770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3433
I20260924 07:11:28.305871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:28.306157 392 external_mini_cluster.cc:1428] Setting key c18f7f1f6d7917d8e9ba323ae882090d
I20260924 07:11:28.307701 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: Bootstrap starting.
I20260924 07:11:28.309965 3501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:28.310701 3501 log.cc:824] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:28.312762 3501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: No bootstrap required, opened a new log
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:28.316484 3501 raft_consensus.cc:348] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260924 07:11:28.316805 3501 raft_consensus.cc:374] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:28.316870 3501 raft_consensus.cc:729] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e4ac6bb2f492da758955126798295, State: Initialized, Role: FOLLOWER
I20260924 07:11:28.317345 3501 consensus_queue.cc:260] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260924 07:11:28.317471 3501 raft_consensus.cc:388] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:28.317528 3501 raft_consensus.cc:482] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:28.317625 3501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:28.318528 3501 raft_consensus.cc:504] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260924 07:11:28.318876 3501 leader_election.cc:302] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b90e4ac6bb2f492da758955126798295; no voters:
I20260924 07:11:28.319195 3501 leader_election.cc:288] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:28.319344 3506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:28.319624 3506 raft_consensus.cc:686] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [term 1 LEADER]: Becoming Leader. State: Replica: b90e4ac6bb2f492da758955126798295, State: Running, Role: LEADER
I20260924 07:11:28.319972 3506 consensus_queue.cc:237] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260924 07:11:28.320559 3501 sys_catalog.cc:564] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:28.320559 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:28.307660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.0.98.62:34723)
0924 07:11:28.308152 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:28.308168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:28.308207 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:11:28.308900 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:28.308919 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:28.309279 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:28.309596 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:28.310405 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.311660 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.312542 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.313070 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.313222 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:28.315753 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:28.315760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:28.315775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:28.315805 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:28.317560 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.318002 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.318008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.318013 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.318126 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.318385 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.318389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.318391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.318836 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:28.319009 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:28.319298 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 07:11:28.319649 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260924 07:11:28.321244 3506 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b90e4ac6bb2f492da758955126798295. Latest consensus state: current_term: 1 leader_uuid: "b90e4ac6bb2f492da758955126798295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260924 07:11:28.321367 3506 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:28.322695 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b90e4ac6bb2f492da758955126798295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b90e4ac6bb2f492da758955126798295" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260924 07:11:28.323312 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295 [sys.catalog]: This master's current role is: LEADER
W20260924 07:11:28.325282 3521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:11:28.325343 3521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:11:28.325430 3513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:28.326081 3513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:28.332187 3513 catalog_manager.cc:1347] Generated new cluster ID: 6a9de7c9665c43ee88019cbcf32e4c22
I20260924 07:11:28.332269 3513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:28.361737 3513 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:28.363130 3513 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:28.369244 3513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b90e4ac6bb2f492da758955126798295: Generated new TSK 0
I20260924 07:11:28.370100 3513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:28.402839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:28.514345 3527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:28.514634 3527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:28.514694 3527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:28.518769 3527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:28.518837 3527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:28.518918 3527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:28.524489 3527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:28.525564 3527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:28.526585 3527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:28.536460 3535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.536463 3533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.536501 3527 server_base.cc:1034] running on GCE node
W20260924 07:11:28.536463 3532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.537143 3527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:28.537851 3527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:28.539044 3527 hybrid_clock.cc:648] HybridClock initialized: now 1790233888539027 us; error 31 us; skew 500 ppm
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:28.542312 3527 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:28.543493 3527 webserver.cc:460] Webserver started at http://127.0.98.1:40625/ using document root <none> and password file <none>
I20260924 07:11:28.544075 3527 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:28.544137 3527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:28.544334 3527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:28.546443 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "b28673487ca24204a0b6bc2490db4430"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "8c02e6f9054658a56b8eec0c51901999"
server_key_iv: "3b2cac642750a3a012b326015ea5cc2b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.546998 3527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b28673487ca24204a0b6bc2490db4430"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "8c02e6f9054658a56b8eec0c51901999"
server_key_iv: "3b2cac642750a3a012b326015ea5cc2b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.551162 3527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 07:11:28.553858 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.555112 3527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:28.555229 3527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "b28673487ca24204a0b6bc2490db4430"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "8c02e6f9054658a56b8eec0c51901999"
server_key_iv: "3b2cac642750a3a012b326015ea5cc2b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.555335 3527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:28.575608 3527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:28.579036 3527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:28.579218 3527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:28.579861 3527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:28.580729 3527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:28.580777 3527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.580830 3527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:28.580858 3527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.589303 3527 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42923
I20260924 07:11:28.589347 3655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42923 every 8 connection(s)
I20260924 07:11:28.590204 3527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:28.599810 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3527
I20260924 07:11:28.599910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:28.600201 392 external_mini_cluster.cc:1428] Setting key a628ccd32f6c728f41a4c6267bba33b3
I20260924 07:11:28.604283 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:28.591988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55059 (local address 127.0.98.62:34723)
0924 07:11:28.592159 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:28.592163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:28.592968 (+ 805us) server_negotiation.cc:407] Connection header received
0924 07:11:28.594301 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:28.594304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:28.594351 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:28.594426 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:28.595853 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.596392 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.597511 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.597793 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.597884 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:28.600480 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:28.600484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:28.600490 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:28.600519 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:28.602085 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.602564 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.602567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.602569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.602657 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.602961 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.602964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.602966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.603172 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:28.603326 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:28.604016 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 07:11:28.604135 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 07:11:28.605384 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:28.592351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.1:55059)
0924 07:11:28.592808 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:28.592854 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:28.594089 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:28.594638 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:28.594646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:28.595125 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:28.595708 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:28.595723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.596495 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.596498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:28.597380 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:28.597388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.597927 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.597929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:28.598065 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:28.598640 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:28.598656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:28.600297 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:28.602207 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:28.602212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:28.602225 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:28.602469 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:28.602746 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:28.602748 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:28.602750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:28.602878 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:28.603356 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:28.603363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:28.603732 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 07:11:28.603951 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 07:11:28.604327 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260924 07:11:28.606756 3656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260924 07:11:28.607030 3656 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:28.607687 3656 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260924 07:11:28.609563 3461 ts_manager.cc:194] Registered new tserver with Master: b28673487ca24204a0b6bc2490db4430 (127.0.98.1:42923)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:28.610805 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:28.630861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:28.742573 3661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:28.742877 3661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:28.742921 3661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:28.747133 3661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:28.747208 3661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:28.747273 3661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:28.752645 3661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:28.753698 3661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:28.754652 3661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:28.764652 3667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.764652 3666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.764647 3669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.764840 3661 server_base.cc:1034] running on GCE node
I20260924 07:11:28.765308 3661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:28.766057 3661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:28.767266 3661 hybrid_clock.cc:648] HybridClock initialized: now 1790233888767248 us; error 33 us; skew 500 ppm
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:28.773926 3661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:28.774896 3661 webserver.cc:460] Webserver started at http://127.0.98.2:44303/ using document root <none> and password file <none>
I20260924 07:11:28.775496 3661 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:28.775579 3661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:28.775810 3661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:28.777889 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "d339f44a1e4f4110bec8d5fe2362b528"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "97fd030a938e13bbbd979cefd90724d8"
server_key_iv: "ceaae38ebf4d3ced2dc781a0837b7b9b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.778417 3661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d339f44a1e4f4110bec8d5fe2362b528"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "97fd030a938e13bbbd979cefd90724d8"
server_key_iv: "ceaae38ebf4d3ced2dc781a0837b7b9b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.782198 3661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:28.784778 3676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.786195 3661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:28.786304 3661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "d339f44a1e4f4110bec8d5fe2362b528"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "97fd030a938e13bbbd979cefd90724d8"
server_key_iv: "ceaae38ebf4d3ced2dc781a0837b7b9b"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.786401 3661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:28.801522 3661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:28.804793 3661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:28.804946 3661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:28.805495 3661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:28.806298 3661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:28.806342 3661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.806380 3661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:28.806394 3661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:28.815049 3661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42319
I20260924 07:11:28.815073 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42319 every 8 connection(s)
I20260924 07:11:28.816047 3661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:28.817325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3661
I20260924 07:11:28.817453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:28.817755 392 external_mini_cluster.cc:1428] Setting key bdd72920b9a4399197bdb6c5f32d0ef2
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:28.832435 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:28.818261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42893 (local address 127.0.98.62:34723)
0924 07:11:28.818396 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:28.818400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:28.819615 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 07:11:28.821440 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:28.821443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:28.821492 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:28.821622 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:28.823051 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.823867 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.825179 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.825556 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.825673 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:28.828447 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:28.828451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:28.828454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:28.828482 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:28.830036 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.830610 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.830613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.830614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.830770 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:28.831181 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:28.831184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:28.831186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:28.831469 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:28.831569 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:28.832116 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 07:11:28.832270 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:11:28.833348 3792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:28.818845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.2:42893)
0924 07:11:28.819459 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:28.819514 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:28.821116 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:28.821840 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:28.821851 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:28.822352 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:28.822887 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:28.822902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.824022 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.824025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:28.825029 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:28.825039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:28.825746 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:28.825750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:28.825920 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:28.826509 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:28.826526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:28.828258 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:28.830199 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:28.830222 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:28.830237 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:28.830500 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:28.830876 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:28.830878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:28.830880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:28.831045 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:28.831594 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:28.831601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:28.831870 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 07:11:28.832053 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 07:11:28.832447 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":161,"threads_started":1}
I20260924 07:11:28.834873 3790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260924 07:11:28.835136 3790 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:28.835752 3790 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260924 07:11:28.836985 3461 ts_manager.cc:194] Registered new tserver with Master: d339f44a1e4f4110bec8d5fe2362b528 (127.0.98.2:42319)
I20260924 07:11:28.837766 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:28.851620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:28.961861 3795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:28.962108 3795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:28.962152 3795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:28.966209 3795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:28.966271 3795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:28.966336 3795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:28.971704 3795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:28.972724 3795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:28.973728 3795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:28.983335 3803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:28.983325 3800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.983556 3795 server_base.cc:1034] running on GCE node
W20260924 07:11:28.983707 3801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:28.984138 3795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:28.984819 3795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:28.985980 3795 hybrid_clock.cc:648] HybridClock initialized: now 1790233888985950 us; error 41 us; skew 500 ppm
Sep 24 07:11:28 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:28.988737 3795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:28.989609 3795 webserver.cc:460] Webserver started at http://127.0.98.3:45631/ using document root <none> and password file <none>
I20260924 07:11:28.990236 3795 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:28.990303 3795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:28.990494 3795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:28.992544 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "beb2025655d545b2a872b4f76c9f775f"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "144ddbf32fcf2d1dc3d1c7db8e71cace"
server_key_iv: "104b078ee225c871745d059462d1418d"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.993048 3795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "beb2025655d545b2a872b4f76c9f775f"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "144ddbf32fcf2d1dc3d1c7db8e71cace"
server_key_iv: "104b078ee225c871745d059462d1418d"
server_key_version: "encryptionkey@0"
I20260924 07:11:28.996806 3795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:28.999275 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:29.000650 3795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:29.000831 3795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "beb2025655d545b2a872b4f76c9f775f"
format_stamp: "Formatted at 2026-09-24 07:11:28 on dist-test-slave-cnv3"
server_key: "144ddbf32fcf2d1dc3d1c7db8e71cace"
server_key_iv: "104b078ee225c871745d059462d1418d"
server_key_version: "encryptionkey@0"
I20260924 07:11:29.000936 3795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:29.038403 3795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:29.041988 3795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:29.042196 3795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:29.042809 3795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:29.043934 3795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:29.043996 3795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:29.044052 3795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:29.044085 3795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:29.052703 3795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37271
I20260924 07:11:29.052740 3923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37271 every 8 connection(s)
I20260924 07:11:29.053593 3795 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:29.058487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3795
I20260924 07:11:29.058632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:29.058955 392 external_mini_cluster.cc:1428] Setting key 3e67f1d905e50737e9fbedf1a45be0e4
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:29.068516 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:29.055467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53997 (local address 127.0.98.62:34723)
0924 07:11:29.055605 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:29.055609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:29.056495 (+ 886us) server_negotiation.cc:407] Connection header received
0924 07:11:29.057897 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:29.057900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:29.057948 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:29.058027 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:29.059593 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.060224 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.061773 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.062046 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.062141 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:29.064922 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:29.064925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:29.064927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:29.064957 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:29.066275 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:29.066858 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:29.066862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:29.066864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:29.066958 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:29.067341 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:29.067344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:29.067345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:29.067624 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:29.067709 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:29.068284 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 07:11:29.068395 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:11:29.069396 3926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:29.055826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.3:53997)
0924 07:11:29.056331 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:29.056377 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:29.057669 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:29.058231 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:29.058238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:29.058699 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:29.059455 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:29.059476 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.060377 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.060381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:29.061583 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:29.061599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.062221 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.062225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:29.062378 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:29.063029 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:29.063047 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:29.064755 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:29.066402 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:29.066408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:29.066421 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:29.066713 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:29.067061 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:29.067064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:29.067066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:29.067212 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:29.067789 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:29.067796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:29.068059 (+ 263us) client_negotiation.cc:769] Sending connection context
0924 07:11:29.068223 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 07:11:29.068507 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":116,"threads_started":1}
I20260924 07:11:29.070555 3924 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260924 07:11:29.070784 3924 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:29.071271 3924 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260924 07:11:29.072216 3461 ts_manager.cc:194] Registered new tserver with Master: beb2025655d545b2a872b4f76c9f775f (127.0.98.3:37271)
I20260924 07:11:29.072875 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53997
I20260924 07:11:29.073037 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233889, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233889, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:29.097129 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:29.089147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.0.98.62:34723)
0924 07:11:29.089314 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:29.089318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:29.089421 (+ 103us) server_negotiation.cc:407] Connection header received
0924 07:11:29.089542 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:29.089545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:29.089587 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:29.089655 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:29.090375 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.090920 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.091842 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:29.092079 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:29.092164 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:29.094558 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:29.094561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:29.094563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:29.094586 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:29.095818 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:29.096169 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:29.096171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:29.096173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:29.096241 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:29.096434 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:29.096437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:29.096438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:29.096638 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:29.096734 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:29.096922 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 07:11:29.097012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 07:11:29.097523 3461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53344
I20260924 07:11:29.098488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3527
I20260924 07:11:29.103508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3661
I20260924 07:11:29.107890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3795
I20260924 07:11:29.112147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3433
2026-09-24T07:11:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2088 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3936](info): set up 2 sockets
Sep 24 07:11:29 dist-test-slave-cnv3 krb5kdc[3936](info): commencing operation
krb5kdc: starting...
W20260924 07:11:30.079262 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:30.204845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:30.315730 3949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:30.316033 3949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:30.316092 3949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:30.320266 3949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:30.320338 3949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:30.320364 3949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:30.320389 3949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:30.320417 3949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:30.326094 3949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:30.327303 3949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:30.328450 3949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:30.337204 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:30.337204 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:30.337236 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:30.337548 3949 server_base.cc:1034] running on GCE node
I20260924 07:11:30.338296 3949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:30.339494 3949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:30.340689 3949 hybrid_clock.cc:648] HybridClock initialized: now 1790233890340677 us; error 28 us; skew 500 ppm
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:30.343776 3949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:30.344738 3949 webserver.cc:460] Webserver started at http://127.0.98.62:33677/ using document root <none> and password file <none>
I20260924 07:11:30.345278 3949 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:30.345338 3949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:30.345522 3949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:30.347860 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "ed79accc150f451eae70790203dbc710"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "88a2b809c9ba08a7506fa8485593eea5"
server_key_iv: "578b381a2121f61abc6bae1c428e3bb9"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.348498 3949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed79accc150f451eae70790203dbc710"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "88a2b809c9ba08a7506fa8485593eea5"
server_key_iv: "578b381a2121f61abc6bae1c428e3bb9"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.352589 3949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 07:11:30.355222 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.356575 3949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:30.356680 3949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "ed79accc150f451eae70790203dbc710"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "88a2b809c9ba08a7506fa8485593eea5"
server_key_iv: "578b381a2121f61abc6bae1c428e3bb9"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.356793 3949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:30.376384 3949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:30.379685 3949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:30.379863 3949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:30.386832 3949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40325
I20260924 07:11:30.386855 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40325 every 8 connection(s)
I20260924 07:11:30.387840 3949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:30.390616 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:30.391587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 3949
I20260924 07:11:30.391701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:30.392001 392 external_mini_cluster.cc:1428] Setting key a2889223e390228d7a4582627fb9c48f
I20260924 07:11:30.397294 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710: Bootstrap starting.
I20260924 07:11:30.399964 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:30.400879 4017 log.cc:824] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:30.403156 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710: No bootstrap required, opened a new log
I20260924 07:11:30.406893 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260924 07:11:30.407130 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:30.407163 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed79accc150f451eae70790203dbc710, State: Initialized, Role: FOLLOWER
I20260924 07:11:30.407297 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:30.393596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.98.62:40325)
0924 07:11:30.394055 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:30.394069 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:30.394353 (+ 284us) server_negotiation.cc:407] Connection header received
0924 07:11:30.395180 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:30.395199 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:30.395620 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:30.395906 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:30.396769 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.397824 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.399008 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.399572 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.399704 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:30.402554 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:30.402559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:30.402574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:30.402601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:30.404330 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.404756 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.404762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.404768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.404899 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.405161 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.405164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.405166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.405565 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:30.405732 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:30.406190 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 07:11:30.406467 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":212,"threads_started":1}
I20260924 07:11:30.407770 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260924 07:11:30.407984 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:30.408051 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:30.408150 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:30.409037 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260924 07:11:30.409371 4017 leader_election.cc:302] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed79accc150f451eae70790203dbc710; no voters:
I20260924 07:11:30.409693 4017 leader_election.cc:288] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:30.409792 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:30.410090 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [term 1 LEADER]: Becoming Leader. State: Replica: ed79accc150f451eae70790203dbc710, State: Running, Role: LEADER
I20260924 07:11:30.410414 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260924 07:11:30.410871 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:30.412336 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed79accc150f451eae70790203dbc710. Latest consensus state: current_term: 1 leader_uuid: "ed79accc150f451eae70790203dbc710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } } }
I20260924 07:11:30.412485 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:30.412319 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed79accc150f451eae70790203dbc710" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed79accc150f451eae70790203dbc710" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } } }
I20260924 07:11:30.412794 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:30.412853 4031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:30.415870 4031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:30.421984 4031 catalog_manager.cc:1347] Generated new cluster ID: 5676ed5b7bf44eb0866e7c4d64740fc7
I20260924 07:11:30.422056 4031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:30.436499 4031 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:30.437474 4031 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:30.445605 4031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed79accc150f451eae70790203dbc710: Generated new TSK 0
I20260924 07:11:30.446506 4031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:30.483135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:30.596241 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:30.596524 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:30.596585 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:30.600888 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:30.600953 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:30.601037 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:30.606465 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:30.607576 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:30.608583 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:30.618247 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:30.618248 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:30.618352 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:30.618490 4043 server_base.cc:1034] running on GCE node
I20260924 07:11:30.618950 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:30.619674 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:30.620841 4043 hybrid_clock.cc:648] HybridClock initialized: now 1790233890620830 us; error 28 us; skew 500 ppm
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:30.623656 4043 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:30.624601 4043 webserver.cc:460] Webserver started at http://127.0.98.1:37045/ using document root <none> and password file <none>
I20260924 07:11:30.625151 4043 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:30.625213 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:30.625414 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:30.627534 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "d43e74746cfd4707a0d0fbc8a040c6c0"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "55eafd5e63e5251c8b4ebf77a93e9e12"
server_key_iv: "872df8e5e7cd47896fb722253cd1fc97"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.628075 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d43e74746cfd4707a0d0fbc8a040c6c0"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "55eafd5e63e5251c8b4ebf77a93e9e12"
server_key_iv: "872df8e5e7cd47896fb722253cd1fc97"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.632161 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:30.634675 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.636104 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:30.636206 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "d43e74746cfd4707a0d0fbc8a040c6c0"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "55eafd5e63e5251c8b4ebf77a93e9e12"
server_key_iv: "872df8e5e7cd47896fb722253cd1fc97"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.636302 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:30.649042 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:30.652689 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:30.652864 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:30.653448 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:30.654397 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:30.654460 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.654512 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:30.654531 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.663532 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41311
I20260924 07:11:30.663565 4171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41311 every 8 connection(s)
I20260924 07:11:30.664479 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:30.670230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4043
I20260924 07:11:30.670353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:30.670629 392 external_mini_cluster.cc:1428] Setting key 7fc0d77449cf0f36a164955d8314b438
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:30.680756 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:30.666512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40681 (local address 127.0.98.62:40325)
0924 07:11:30.666681 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:30.666686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:30.667601 (+ 915us) server_negotiation.cc:407] Connection header received
0924 07:11:30.669064 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:30.669067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:30.669118 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:30.669198 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:30.670607 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.671305 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.673131 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.673479 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.673600 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:30.676650 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:30.676653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:30.676655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:30.676689 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:30.678330 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.678919 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.678924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.678925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.679026 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.679460 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.679464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.679466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.679718 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:30.679826 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:30.680418 (+ 592us) server_negotiation.cc:299] Negotiation successful
0924 07:11:30.680569 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:11:30.681586 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:30.666932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40325 (local address 127.0.98.1:40681)
0924 07:11:30.667447 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:30.667498 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:30.668835 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:30.669423 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:30.669432 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:30.669836 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:30.670443 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:30.670461 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.671882 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.671886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:30.672984 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:30.672996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.673708 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.673712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:30.673937 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:30.674639 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:30.674662 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:30.676418 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:30.678496 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:30.678502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:30.678518 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:30.678798 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:30.679152 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:30.679154 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:30.679156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:30.679316 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:30.679834 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:30.679840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:30.680140 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 07:11:30.680308 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 07:11:30.680659 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260924 07:11:30.683049 4172 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:30.683323 4172 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:30.683873 4172 heartbeater.cc:511] Master 127.0.98.62:40325 requested a full tablet report, sending...
I20260924 07:11:30.685561 3980 ts_manager.cc:194] Registered new tserver with Master: d43e74746cfd4707a0d0fbc8a040c6c0 (127.0.98.1:41311)
I20260924 07:11:30.686728 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:30.699648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:30.815192 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:30.815480 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:30.815526 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:30.819643 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:30.819715 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:30.819777 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:30.825292 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:30.826370 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:30.827422 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:30.837709 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:30.837704 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:30.837913 4177 server_base.cc:1034] running on GCE node
W20260924 07:11:30.837704 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:30.838546 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:30.839260 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:30.840449 4177 hybrid_clock.cc:648] HybridClock initialized: now 1790233890840411 us; error 50 us; skew 500 ppm
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:30.843532 4177 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:30.844540 4177 webserver.cc:460] Webserver started at http://127.0.98.2:32891/ using document root <none> and password file <none>
I20260924 07:11:30.845153 4177 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:30.845229 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:30.845624 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:30.847975 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "526771343b7b464684cd73e4fab0145b"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "e0a0f76370f13ca2fb0ec766bac30cc2"
server_key_iv: "75c34cd164ffde3c2cb4bfe981310669"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.848727 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "526771343b7b464684cd73e4fab0145b"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "e0a0f76370f13ca2fb0ec766bac30cc2"
server_key_iv: "75c34cd164ffde3c2cb4bfe981310669"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.852983 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:30.855849 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.857326 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:30.857450 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "526771343b7b464684cd73e4fab0145b"
format_stamp: "Formatted at 2026-09-24 07:11:30 on dist-test-slave-cnv3"
server_key: "e0a0f76370f13ca2fb0ec766bac30cc2"
server_key_iv: "75c34cd164ffde3c2cb4bfe981310669"
server_key_version: "encryptionkey@0"
I20260924 07:11:30.857568 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:30.877209 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:30.880589 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:30.880776 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:30.881386 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:30.882302 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:30.882351 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.882405 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:30.882436 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:30.890981 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33603
I20260924 07:11:30.891085 4305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33603 every 8 connection(s)
I20260924 07:11:30.891942 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:30.896507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4177
I20260924 07:11:30.896634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:30.896867 392 external_mini_cluster.cc:1428] Setting key ca8add495adb1688d124ed4c90e926e8
Sep 24 07:11:30 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:30.906015 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:30.893781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43665 (local address 127.0.98.62:40325)
0924 07:11:30.893932 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:30.893935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:30.894816 (+ 881us) server_negotiation.cc:407] Connection header received
0924 07:11:30.896276 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:30.896280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:30.896338 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:30.896410 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:30.897659 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.898184 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.899317 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.899574 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.899662 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:30.902320 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:30.902324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:30.902327 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:30.902355 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:30.903891 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.904416 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.904420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.904422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.904518 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:30.904846 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:30.904849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:30.904851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:30.905087 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:30.905191 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:30.905720 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 07:11:30.905835 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:11:30.906860 4308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:30.894165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40325 (local address 127.0.98.2:43665)
0924 07:11:30.894651 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:30.894696 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:30.896058 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:30.896607 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:30.896614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:30.897020 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:30.897518 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:30.897532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.898301 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.898304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:30.899193 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:30.899201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:30.899708 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:30.899712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:30.899868 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:30.900403 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:30.900419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:30.902113 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:30.904027 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:30.904033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:30.904045 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:30.904314 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:30.904614 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:30.904617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:30.904618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:30.904754 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:30.905211 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:30.905216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:30.905473 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:11:30.905642 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:11:30.905916 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260924 07:11:30.908165 4306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40325
I20260924 07:11:30.908396 4306 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:30.908946 4306 heartbeater.cc:511] Master 127.0.98.62:40325 requested a full tablet report, sending...
I20260924 07:11:30.910256 3980 ts_manager.cc:194] Registered new tserver with Master: 526771343b7b464684cd73e4fab0145b (127.0.98.2:33603)
I20260924 07:11:30.910902 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:30.926614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:31.038408 4311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:31.038700 4311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:31.038761 4311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:31.043157 4311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:31.043233 4311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:31.043320 4311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:31.048928 4311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:31.050081 4311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:31.051149 4311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:31.061077 4319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:31.061172 4317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:31.061075 4316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:31.061419 4311 server_base.cc:1034] running on GCE node
I20260924 07:11:31.061866 4311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:31.062502 4311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:31.063681 4311 hybrid_clock.cc:648] HybridClock initialized: now 1790233891063662 us; error 35 us; skew 500 ppm
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:31.066617 4311 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:31.067696 4311 webserver.cc:460] Webserver started at http://127.0.98.3:33617/ using document root <none> and password file <none>
I20260924 07:11:31.068266 4311 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:31.068325 4311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:31.068521 4311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:31.070633 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "77f43effa12c4f06a96d3b20b79fc2ca"
format_stamp: "Formatted at 2026-09-24 07:11:31 on dist-test-slave-cnv3"
server_key: "e71a907a96ad452038049adbd34870e7"
server_key_iv: "5ec1dbad3fa25a7c84d7eb0ec111631e"
server_key_version: "encryptionkey@0"
I20260924 07:11:31.071201 4311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77f43effa12c4f06a96d3b20b79fc2ca"
format_stamp: "Formatted at 2026-09-24 07:11:31 on dist-test-slave-cnv3"
server_key: "e71a907a96ad452038049adbd34870e7"
server_key_iv: "5ec1dbad3fa25a7c84d7eb0ec111631e"
server_key_version: "encryptionkey@0"
I20260924 07:11:31.075261 4311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:11:31.077870 4326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:31.079262 4311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:31.079366 4311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "77f43effa12c4f06a96d3b20b79fc2ca"
format_stamp: "Formatted at 2026-09-24 07:11:31 on dist-test-slave-cnv3"
server_key: "e71a907a96ad452038049adbd34870e7"
server_key_iv: "5ec1dbad3fa25a7c84d7eb0ec111631e"
server_key_version: "encryptionkey@0"
I20260924 07:11:31.079510 4311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:31.088573 4311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:31.091904 4311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:31.092069 4311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:31.092695 4311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:31.093562 4311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:31.093614 4311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:31.093653 4311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:31.093667 4311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:31.102221 4311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40381
I20260924 07:11:31.102283 4439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40381 every 8 connection(s)
I20260924 07:11:31.103127 4311 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:31.112934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4311
I20260924 07:11:31.113041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:31.113278 392 external_mini_cluster.cc:1428] Setting key cd30ba50bc876f0a122eb0f1f9625acd
I20260924 07:11:31.117506 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:31.105070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52487 (local address 127.0.98.62:40325)
0924 07:11:31.105229 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:31.105234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:31.106168 (+ 934us) server_negotiation.cc:407] Connection header received
0924 07:11:31.107568 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:31.107571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:31.107620 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:31.107699 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:31.109041 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.109561 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.110900 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.111136 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.111256 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:31.113878 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:31.113881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:31.113883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:31.113908 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:31.115611 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.116094 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.116097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.116098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.116181 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.116513 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.116517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.116519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.116708 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:31.116793 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:31.117267 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 07:11:31.117379 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:11:31.118346 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:31.105503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40325 (local address 127.0.98.3:52487)
0924 07:11:31.105957 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:31.106049 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:31.107287 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:31.107863 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:31.107870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:31.108292 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:31.108799 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:31.108812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.109700 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.109703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:31.110773 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:31.110784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.111355 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.111358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:31.111534 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:31.112121 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:31.112138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:31.113713 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:31.115711 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:31.115716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:31.115730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:31.115988 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:31.116261 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:31.116263 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:31.116265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:31.116399 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:31.116794 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:31.116799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:31.117043 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 07:11:31.117201 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 07:11:31.117460 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260924 07:11:31.119755 4440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40325
I20260924 07:11:31.120018 4440 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:31.120580 4440 heartbeater.cc:511] Master 127.0.98.62:40325 requested a full tablet report, sending...
I20260924 07:11:31.121660 3980 ts_manager.cc:194] Registered new tserver with Master: 77f43effa12c4f06a96d3b20b79fc2ca (127.0.98.3:40381)
I20260924 07:11:31.122231 3980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52487
I20260924 07:11:31.127292 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[3936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:31.153821 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:31.143403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.0.98.1:41311)
0924 07:11:31.143746 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:31.143761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:31.143784 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:11:31.143835 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:31.143840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:31.144039 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:31.144160 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:31.144893 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.145742 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.146697 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.147167 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.147266 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:31.149932 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:31.149944 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:31.149959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:31.150042 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:31.151851 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.152341 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.152348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.152349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.152434 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.152717 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.152720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.152721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.152986 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:31.153253 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:31.153461 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:11:31.153603 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[3936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:31.165266 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:31.157088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.98.62:40325)
0924 07:11:31.157225 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:31.157229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:31.157283 (+ 54us) server_negotiation.cc:407] Connection header received
0924 07:11:31.157420 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:31.157423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:31.157469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:31.157543 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:31.158136 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.158667 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.159511 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:31.159725 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:31.159807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:31.162233 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:31.162235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:31.162238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:31.162264 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:31.163640 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.164024 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.164028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.164030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.164104 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:31.164352 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:31.164355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:31.164357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:31.164628 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:31.164749 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:31.164973 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 07:11:31.165103 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 07:11:31.165653 3980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55672
I20260924 07:11:31.166406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4043
I20260924 07:11:31.171329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4177
I20260924 07:11:31.175984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4311
I20260924 07:11:31.180383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 3949
2026-09-24T07:11:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2067 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[4452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[4452](info): set up 2 sockets
Sep 24 07:11:31 dist-test-slave-cnv3 krb5kdc[4452](info): commencing operation
krb5kdc: starting...
W20260924 07:11:32.151506 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:32.278604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:32.388734 4465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:32.389034 4465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:32.389093 4465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:32.393215 4465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:32.393297 4465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:32.393323 4465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:32.393344 4465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:32.393365 4465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:32.399112 4465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:32.400413 4465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:32.401530 4465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:32.409902 4471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:32.409902 4473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.410032 4465 server_base.cc:1034] running on GCE node
W20260924 07:11:32.409960 4470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.410869 4465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:32.412041 4465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:32.413220 4465 hybrid_clock.cc:648] HybridClock initialized: now 1790233892413199 us; error 40 us; skew 500 ppm
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:32.416019 4465 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:32.416747 4465 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:32.416816 4465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:32.417004 4465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:32.419052 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "44f51cfa68d544c389f17cbf2d3ba5f4"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "e6687e1b34bcc26905ca5800a9a7c50b"
server_key_iv: "579d5aeac5e976b079375653b6db853a"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.419631 4465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "44f51cfa68d544c389f17cbf2d3ba5f4"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "e6687e1b34bcc26905ca5800a9a7c50b"
server_key_iv: "579d5aeac5e976b079375653b6db853a"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.423316 4465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:11:32.425789 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.427021 4465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:11:32.427145 4465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "44f51cfa68d544c389f17cbf2d3ba5f4"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "e6687e1b34bcc26905ca5800a9a7c50b"
server_key_iv: "579d5aeac5e976b079375653b6db853a"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.427242 4465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:32.447551 4465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:32.450954 4465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:32.451150 4465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:32.458606 4465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43853
I20260924 07:11:32.458849 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43853 every 8 connection(s)
I20260924 07:11:32.459306 4465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:32.462217 4532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:32.465097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4465
I20260924 07:11:32.465222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:32.465500 392 external_mini_cluster.cc:1428] Setting key cc4254311e96e8432fe0722a838def21
I20260924 07:11:32.468207 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: Bootstrap starting.
I20260924 07:11:32.470691 4532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:32.471616 4532 log.cc:824] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:32.473991 4532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: No bootstrap required, opened a new log
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:32.477221 4532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260924 07:11:32.477452 4532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:32.477520 4532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f51cfa68d544c389f17cbf2d3ba5f4, State: Initialized, Role: FOLLOWER
I20260924 07:11:32.477948 4532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260924 07:11:32.478114 4532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:32.478193 4532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:32.478298 4532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:32.479308 4532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260924 07:11:32.479722 4532 leader_election.cc:302] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f51cfa68d544c389f17cbf2d3ba5f4; no voters:
I20260924 07:11:32.480047 4532 leader_election.cc:288] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:32.480157 4537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:32.480357 4537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 44f51cfa68d544c389f17cbf2d3ba5f4, State: Running, Role: LEADER
I20260924 07:11:32.480744 4537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260924 07:11:32.480927 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:32.466894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.0.98.62:43853)
0924 07:11:32.467299 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:32.467314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:32.467350 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:32.468009 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:32.468026 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:32.468408 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:32.468694 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:32.469521 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.470548 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.471806 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.472507 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.472642 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:32.475484 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:32.475492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:32.475518 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:32.475551 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:32.477823 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.478307 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.478312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.478317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.478453 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.478735 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.478737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.478739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.479123 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:32.479335 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:32.479596 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 07:11:32.479892 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260924 07:11:32.481341 4532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:32.482026 4537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f51cfa68d544c389f17cbf2d3ba5f4. Latest consensus state: current_term: 1 leader_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260924 07:11:32.482232 4537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:32.482442 4538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f51cfa68d544c389f17cbf2d3ba5f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260924 07:11:32.482887 4538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:32.483975 4545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:32.486661 4545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:32.492297 4545 catalog_manager.cc:1347] Generated new cluster ID: d0b18d636bfa4b55989d51e33bb2a83a
I20260924 07:11:32.492357 4545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:32.504172 4545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:32.505513 4545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:32.518672 4545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: Generated new TSK 0
I20260924 07:11:32.519608 4545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:32.558795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:32.669943 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:32.670233 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:32.670279 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:32.674474 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:32.674547 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:32.674614 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:32.680238 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:32.681432 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:32.682633 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:32.692646 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:32.692625 4563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.692703 4558 server_base.cc:1034] running on GCE node
W20260924 07:11:32.692628 4564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.693387 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:32.694088 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:32.695292 4558 hybrid_clock.cc:648] HybridClock initialized: now 1790233892695278 us; error 27 us; skew 500 ppm
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:32.698195 4558 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:32.699052 4558 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:32.699122 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:32.699347 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:32.701512 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "6344097e897541a8aa6ffd7260f93abb"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "d5700206a3ea75e907fe7cd4a930e91e"
server_key_iv: "8bd746d0b0ba5780036f4917ac2be40d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.702081 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6344097e897541a8aa6ffd7260f93abb"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "d5700206a3ea75e907fe7cd4a930e91e"
server_key_iv: "8bd746d0b0ba5780036f4917ac2be40d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.706053 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 07:11:32.708732 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.710088 4558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:32.710189 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "6344097e897541a8aa6ffd7260f93abb"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "d5700206a3ea75e907fe7cd4a930e91e"
server_key_iv: "8bd746d0b0ba5780036f4917ac2be40d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.710310 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:32.735966 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:32.739025 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:32.739179 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:32.739805 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:32.740682 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:32.740729 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.740772 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:32.740795 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.749222 4558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35473
I20260924 07:11:32.749322 4685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35473 every 8 connection(s)
I20260924 07:11:32.749881 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:32.755255 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4558
I20260924 07:11:32.755407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:32.755681 392 external_mini_cluster.cc:1428] Setting key ff5a282c89c05fc32dd456fe831ac334
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:32.764472 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:32.751789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53777 (local address 127.0.98.62:43853)
0924 07:11:32.751938 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:32.751942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:32.752948 (+ 1006us) server_negotiation.cc:407] Connection header received
0924 07:11:32.754337 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:32.754340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:32.754391 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:32.754470 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:32.755910 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.756529 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.757832 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.758143 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.758239 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:32.761039 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:32.761042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:32.761044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:32.761068 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:32.762433 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.762969 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.762973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.762975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.763050 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.763401 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.763405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.763407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.763587 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:32.763668 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:32.764182 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:11:32.764313 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:32.765321 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:32.752239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43853 (local address 127.0.98.1:53777)
0924 07:11:32.752772 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:32.752822 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:32.754094 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:32.754697 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:32.754705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:32.755104 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:32.755740 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:32.755757 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.756673 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.756676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:32.757689 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:32.757699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.758294 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.758298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:32.758444 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:32.759048 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:32.759069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:32.760846 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:32.762555 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:32.762561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:32.762577 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:32.762847 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:32.763136 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:32.763139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:32.763141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:32.763282 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:32.763688 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:32.763694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:32.763956 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 07:11:32.764127 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:11:32.764463 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":118,"threads_started":1}
I20260924 07:11:32.766593 4686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43853
I20260924 07:11:32.766887 4686 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:32.767349 4686 heartbeater.cc:511] Master 127.0.98.62:43853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:32.768872 4496 ts_manager.cc:194] Registered new tserver with Master: 6344097e897541a8aa6ffd7260f93abb (127.0.98.1:35473)
I20260924 07:11:32.769981 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:32.783808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:32.897823 4691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:32.898095 4691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:32.898139 4691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:32.902288 4691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:32.902361 4691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:32.902423 4691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:32.907861 4691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:32.908942 4691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:32.909967 4691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:32.919863 4699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:32.919863 4696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.919857 4691 server_base.cc:1034] running on GCE node
W20260924 07:11:32.919884 4697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:32.920581 4691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:32.921247 4691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:32.922420 4691 hybrid_clock.cc:648] HybridClock initialized: now 1790233892922402 us; error 37 us; skew 500 ppm
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:32.925397 4691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:32.926138 4691 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:32.926211 4691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:32.926410 4691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:32.928505 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "25f49ecbb1614d819841ccecd83114d6"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "460ada5721018e88683a8e68b690baaa"
server_key_iv: "b2ec5ad44f752f8a1446beacd93a085d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.929050 4691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "25f49ecbb1614d819841ccecd83114d6"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "460ada5721018e88683a8e68b690baaa"
server_key_iv: "b2ec5ad44f752f8a1446beacd93a085d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.932924 4691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:32.935369 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.936717 4691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:32.936829 4691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "25f49ecbb1614d819841ccecd83114d6"
format_stamp: "Formatted at 2026-09-24 07:11:32 on dist-test-slave-cnv3"
server_key: "460ada5721018e88683a8e68b690baaa"
server_key_iv: "b2ec5ad44f752f8a1446beacd93a085d"
server_key_version: "encryptionkey@0"
I20260924 07:11:32.936930 4691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:32.957096 4691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:32.960142 4691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:32.960333 4691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:32.960919 4691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:32.961772 4691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:32.961822 4691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.961875 4691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:32.961905 4691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:32.970511 4691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38121
I20260924 07:11:32.970530 4818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38121 every 8 connection(s)
I20260924 07:11:32.971087 4691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:11:32 dist-test-slave-cnv3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:32.980368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4691
I20260924 07:11:32.980463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:32.980768 392 external_mini_cluster.cc:1428] Setting key 6c20f07d0b2ba4a24210a4429cba9080
I20260924 07:11:32.984845 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:32.972816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46853 (local address 127.0.98.62:43853)
0924 07:11:32.972935 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:32.972938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:32.973810 (+ 872us) server_negotiation.cc:407] Connection header received
0924 07:11:32.975162 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:32.975165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:32.975232 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:32.975301 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:32.976501 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.977016 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.978144 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.978365 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.978455 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:32.981068 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:32.981073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:32.981075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:32.981104 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:32.982650 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.983283 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.983287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.983289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.983387 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:32.983716 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:32.983719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:32.983721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:32.983956 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:32.984051 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:32.984560 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 07:11:32.984681 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:11:32.985797 4821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:32.973201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43853 (local address 127.0.98.2:46853)
0924 07:11:32.973642 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:32.973685 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:32.974951 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:32.975503 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:32.975510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:32.975871 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:32.976368 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:32.976383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.977133 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.977136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:32.978026 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:32.978035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:32.978516 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:32.978521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:32.978676 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:32.979224 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:32.979241 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:32.980882 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:32.982759 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:32.982765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:32.982857 (+ 92us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:32.983187 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:32.983471 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:32.983474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:32.983476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:32.983621 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:32.984063 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:32.984069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:32.984329 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 07:11:32.984494 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 07:11:32.984776 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260924 07:11:32.986939 4819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43853
I20260924 07:11:32.987221 4819 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:32.987790 4819 heartbeater.cc:511] Master 127.0.98.62:43853 requested a full tablet report, sending...
I20260924 07:11:32.988894 4496 ts_manager.cc:194] Registered new tserver with Master: 25f49ecbb1614d819841ccecd83114d6 (127.0.98.2:38121)
I20260924 07:11:32.989454 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46853
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:33.009701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:33.122397 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:33.122673 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:33.122716 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:33.126853 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:33.126920 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:33.126983 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:33.132443 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:33.133538 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:33.134583 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:33.144420 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:33.144439 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:33.144572 4824 server_base.cc:1034] running on GCE node
W20260924 07:11:33.144484 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:33.145128 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:33.145900 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:33.147089 4824 hybrid_clock.cc:648] HybridClock initialized: now 1790233893147059 us; error 42 us; skew 500 ppm
Sep 24 07:11:33 dist-test-slave-cnv3 krb5kdc[4452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:33.150208 4824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:33.151099 4824 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:33.151171 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:33.151435 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:33.153600 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "48863d8e86554237b2554a088d69c8bf"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "d2d381eeeca03841d9d6d4911e239cfc"
server_key_iv: "a1903338856dbf80c3980993b810d766"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.154194 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48863d8e86554237b2554a088d69c8bf"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "d2d381eeeca03841d9d6d4911e239cfc"
server_key_iv: "a1903338856dbf80c3980993b810d766"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.158138 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 07:11:33.160650 4838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:33.161983 4824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:33.162101 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "48863d8e86554237b2554a088d69c8bf"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "d2d381eeeca03841d9d6d4911e239cfc"
server_key_iv: "a1903338856dbf80c3980993b810d766"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.162235 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:33.172740 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:33.176007 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:33.176189 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:33.176808 4824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:33.177635 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:33.177681 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:33.177726 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:33.177757 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:33.186615 4824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37547
I20260924 07:11:33.186683 4951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37547 every 8 connection(s)
I20260924 07:11:33.187197 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:33.196313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4824
I20260924 07:11:33.196463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:33.196741 392 external_mini_cluster.cc:1428] Setting key f8f9abc4c68a126bf3fcfebb3409b6d6
Sep 24 07:11:33 dist-test-slave-cnv3 krb5kdc[4452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:33.201102 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.189032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36933 (local address 127.0.98.62:43853)
0924 07:11:33.189173 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.189177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.190054 (+ 877us) server_negotiation.cc:407] Connection header received
0924 07:11:33.191466 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.191469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.191517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.191588 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:33.192851 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.193430 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.194653 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.194876 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.194959 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.197681 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:33.197685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:33.197688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:33.197716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:33.199148 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:33.199712 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:33.199715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:33.199716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:33.199793 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:33.200090 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:33.200092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:33.200093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:33.200264 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:33.200389 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.200870 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.200978 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:11:33.201972 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.189436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43853 (local address 127.0.98.3:36933)
0924 07:11:33.189893 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.189936 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.191210 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.191799 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.191807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.192191 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:33.192695 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.192709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.193551 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.193554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.194531 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.194541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.195043 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.195046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.195196 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.195818 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:33.195836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:33.197513 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:33.199305 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:33.199310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:33.199323 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:33.199621 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:33.199881 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:33.199884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:33.199886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:33.200021 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:33.200404 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:33.200409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:33.200660 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.200818 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.201101 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260924 07:11:33.203082 4952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43853
I20260924 07:11:33.203325 4952 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:33.203799 4952 heartbeater.cc:511] Master 127.0.98.62:43853 requested a full tablet report, sending...
I20260924 07:11:33.204905 4496 ts_manager.cc:194] Registered new tserver with Master: 48863d8e86554237b2554a088d69c8bf (127.0.98.3:37547)
I20260924 07:11:33.205674 4496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36933
I20260924 07:11:33.210651 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:11:33.219622 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.212553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.0.98.62:43853)
0924 07:11:33.212677 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.212680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.212802 (+ 122us) server_negotiation.cc:407] Connection header received
0924 07:11:33.212879 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.212881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.212925 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.212992 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:33.213684 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.214223 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.215131 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.215473 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.215570 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.216806 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:33.216809 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:33.216811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:33.216835 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:33.217975 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:33.218375 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:33.218378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:33.218379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:33.218455 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:33.218711 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:33.218714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:33.218715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:33.218971 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:33.219087 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.219352 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.219502 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:11:33.222725 4496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:33.225828 4496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:33.240257 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.234563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.0.1:32792)
0924 07:11:33.235002 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.235047 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.235166 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.236268 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.236271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.236296 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:33.236509 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.236516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.237525 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.237528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.238712 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.238720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.239746 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.239750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.239846 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.239879 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.239966 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.240095 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":73,"spinlock_wait_cycles":6400,"thread_start_us":90,"threads_started":1}
I20260924 07:11:33.240298 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.234703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.0.98.2:38121)
0924 07:11:33.235497 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.235508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.235532 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:11:33.235609 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.235615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.235831 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.236360 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:33.236626 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.237404 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.238842 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.239634 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.239738 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.239890 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.240029 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.240132 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":118,"threads_started":1}
I20260924 07:11:33.240556 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.235446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35473 (local address 127.0.0.1:49548)
0924 07:11:33.236042 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.236055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.236133 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.236757 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.236760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.236776 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:33.236848 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.236853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.237945 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.237948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.239082 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.239091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.240193 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.240196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.240275 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.240296 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.240363 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.240428 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260924 07:11:33.240621 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.235853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.0.98.1:35473)
0924 07:11:33.236345 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.236352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.236375 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:11:33.236446 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.236451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.236642 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.236774 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:33.236945 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.237799 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.239224 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.240029 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.240172 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.240313 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.240387 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.240475 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":85,"threads_started":1}
I20260924 07:11:33.241967 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.234613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37547 (local address 127.0.0.1:34028)
0924 07:11:33.235528 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.235540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.235632 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.236253 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.236260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.236290 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:33.236508 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.236516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.238642 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.238645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.239636 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.239645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.241573 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.241577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.241669 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.241689 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.241761 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.241837 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":54,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260924 07:11:33.242010 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.235071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.0.98.3:37547)
0924 07:11:33.235541 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.235552 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.235600 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:11:33.235754 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.235759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.235907 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.236059 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:33.237534 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.238482 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.240884 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.241459 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.241557 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.241696 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.241789 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.241879 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":123,"threads_started":1}
I20260924 07:11:33.243661 4753 tablet_service.cc:1467] Processing CreateTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 (DEFAULT_TABLE table=test-table [id=bc4097aa80274ceea4aa1dbf752f3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:33.243976 4620 tablet_service.cc:1467] Processing CreateTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 (DEFAULT_TABLE table=test-table [id=bc4097aa80274ceea4aa1dbf752f3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:33.244844 4620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d41c1fc8f94d27a3c0a5d64647c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:33.244844 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d41c1fc8f94d27a3c0a5d64647c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:33.244714 4886 tablet_service.cc:1467] Processing CreateTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 (DEFAULT_TABLE table=test-table [id=bc4097aa80274ceea4aa1dbf752f3fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:33.245664 4886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d41c1fc8f94d27a3c0a5d64647c834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:33.252012 4969 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Bootstrap starting.
I20260924 07:11:33.253528 4970 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Bootstrap starting.
I20260924 07:11:33.254638 4969 tablet_bootstrap.cc:654] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:33.255496 4970 tablet_bootstrap.cc:654] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:33.255785 4971 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Bootstrap starting.
I20260924 07:11:33.255803 4969 log.cc:824] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:33.256683 4970 log.cc:824] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:33.257747 4969 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: No bootstrap required, opened a new log
I20260924 07:11:33.257874 4971 tablet_bootstrap.cc:654] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:33.257939 4969 ts_tablet_manager.cc:1397] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:33.258687 4970 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: No bootstrap required, opened a new log
I20260924 07:11:33.258934 4970 ts_tablet_manager.cc:1397] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 07:11:33.259037 4971 log.cc:824] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:33.261113 4971 tablet_bootstrap.cc:492] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: No bootstrap required, opened a new log
I20260924 07:11:33.261315 4971 ts_tablet_manager.cc:1397] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:33.261570 4969 raft_consensus.cc:348] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.261862 4969 raft_consensus.cc:374] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:33.261950 4969 raft_consensus.cc:729] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48863d8e86554237b2554a088d69c8bf, State: Initialized, Role: FOLLOWER
I20260924 07:11:33.262054 4970 raft_consensus.cc:348] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.262421 4970 raft_consensus.cc:374] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:33.262512 4970 raft_consensus.cc:729] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f49ecbb1614d819841ccecd83114d6, State: Initialized, Role: FOLLOWER
I20260924 07:11:33.262456 4969 consensus_queue.cc:260] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.262944 4970 consensus_queue.cc:260] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.263159 4952 heartbeater.cc:503] Master 127.0.98.62:43853 was elected leader, sending a full tablet report...
I20260924 07:11:33.263533 4969 ts_tablet_manager.cc:1428] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:11:33.263991 4970 ts_tablet_manager.cc:1428] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:11:33.264173 4819 heartbeater.cc:503] Master 127.0.98.62:43853 was elected leader, sending a full tablet report...
I20260924 07:11:33.264963 4971 raft_consensus.cc:348] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.265201 4971 raft_consensus.cc:374] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:33.265311 4971 raft_consensus.cc:729] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6344097e897541a8aa6ffd7260f93abb, State: Initialized, Role: FOLLOWER
I20260924 07:11:33.266114 4971 consensus_queue.cc:260] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.266975 4971 ts_tablet_manager.cc:1428] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:11:33.267338 4686 heartbeater.cc:503] Master 127.0.98.62:43853 was elected leader, sending a full tablet report...
W20260924 07:11:33.438138 4953 tablet.cc:2369] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:33.472216 4820 tablet.cc:2369] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:33.501202 4687 tablet.cc:2369] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:33.582808 4975 raft_consensus.cc:482] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:33.583046 4975 raft_consensus.cc:504] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.584368 4975 leader_election.cc:288] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25f49ecbb1614d819841ccecd83114d6 (127.0.98.2:38121), 6344097e897541a8aa6ffd7260f93abb (127.0.98.1:35473)
I20260924 07:11:33.589286 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.584779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51351 (local address 127.0.98.1:35473)
0924 07:11:33.584888 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.584892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.585177 (+ 285us) server_negotiation.cc:407] Connection header received
0924 07:11:33.585257 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.585259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.585308 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.585448 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:33.585681 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.586385 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.587933 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.588577 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.588694 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.588763 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.589072 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.589147 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:11:33.589303 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.584620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.98.3:46117)
0924 07:11:33.584820 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.584837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.584943 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.585208 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.585212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.585226 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:33.585354 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.585360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.586358 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.586361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.587780 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.587798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.588756 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.588761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.588920 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.588962 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.589042 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.589147 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":74}
I20260924 07:11:33.589286 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.584698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46117 (local address 127.0.98.2:38121)
0924 07:11:33.584850 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.584855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.584892 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:11:33.585049 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.585052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.585101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.585181 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:33.585464 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.586218 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.587936 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.588615 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.588704 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.588763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.589072 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.589147 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:33.589408 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.584696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35473 (local address 127.0.98.3:51351)
0924 07:11:33.585080 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.585091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:33.585173 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:33.585480 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:33.585482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:33.585498 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:33.585582 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.585586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.586494 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.586497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.587780 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:33.587793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.588756 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.588761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:33.588915 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.588956 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:11:33.589042 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:11:33.589147 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":95,"thread_start_us":119,"threads_started":1}
I20260924 07:11:33.590068 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d41c1fc8f94d27a3c0a5d64647c834" candidate_uuid: "48863d8e86554237b2554a088d69c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f49ecbb1614d819841ccecd83114d6" is_pre_election: true
I20260924 07:11:33.590252 4640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d41c1fc8f94d27a3c0a5d64647c834" candidate_uuid: "48863d8e86554237b2554a088d69c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6344097e897541a8aa6ffd7260f93abb" is_pre_election: true
I20260924 07:11:33.590384 4773 raft_consensus.cc:2454] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48863d8e86554237b2554a088d69c8bf in term 0.
I20260924 07:11:33.590562 4640 raft_consensus.cc:2454] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48863d8e86554237b2554a088d69c8bf in term 0.
I20260924 07:11:33.590931 4841 leader_election.cc:302] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f49ecbb1614d819841ccecd83114d6, 48863d8e86554237b2554a088d69c8bf; no voters:
I20260924 07:11:33.591180 4975 raft_consensus.cc:2781] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:33.591339 4975 raft_consensus.cc:482] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:33.591424 4975 raft_consensus.cc:3037] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:33.592430 4975 raft_consensus.cc:504] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.592759 4975 leader_election.cc:288] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [CANDIDATE]: Term 1 election: Requested vote from peers 25f49ecbb1614d819841ccecd83114d6 (127.0.98.2:38121), 6344097e897541a8aa6ffd7260f93abb (127.0.98.1:35473)
I20260924 07:11:33.593212 4640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d41c1fc8f94d27a3c0a5d64647c834" candidate_uuid: "48863d8e86554237b2554a088d69c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6344097e897541a8aa6ffd7260f93abb"
I20260924 07:11:33.593324 4640 raft_consensus.cc:3037] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:33.594309 4640 raft_consensus.cc:2454] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48863d8e86554237b2554a088d69c8bf in term 1.
I20260924 07:11:33.594540 4773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d41c1fc8f94d27a3c0a5d64647c834" candidate_uuid: "48863d8e86554237b2554a088d69c8bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25f49ecbb1614d819841ccecd83114d6"
I20260924 07:11:33.594612 4839 leader_election.cc:302] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48863d8e86554237b2554a088d69c8bf, 6344097e897541a8aa6ffd7260f93abb; no voters:
I20260924 07:11:33.594641 4773 raft_consensus.cc:3037] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:33.594781 4975 raft_consensus.cc:2781] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:33.595012 4975 raft_consensus.cc:686] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 1 LEADER]: Becoming Leader. State: Replica: 48863d8e86554237b2554a088d69c8bf, State: Running, Role: LEADER
I20260924 07:11:33.595414 4975 consensus_queue.cc:237] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } }
I20260924 07:11:33.595896 4773 raft_consensus.cc:2454] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48863d8e86554237b2554a088d69c8bf in term 1.
I20260924 07:11:33.598820 4496 catalog_manager.cc:5495] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 48863d8e86554237b2554a088d69c8bf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "48863d8e86554237b2554a088d69c8bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48863d8e86554237b2554a088d69c8bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:33.695218 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.691647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.0.98.3:37547)
0924 07:11:33.691771 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.691774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.691787 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:33.691895 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.691898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.691952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.692039 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:33.692291 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.692846 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.693685 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:33.693917 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:33.694001 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:33.694356 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:33.694792 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:33.694895 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.695000 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.695053 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:33.701295 4773 raft_consensus.cc:1261] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 1 FOLLOWER]: Refusing update from remote peer 48863d8e86554237b2554a088d69c8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:33.701298 4640 raft_consensus.cc:1261] T f7d41c1fc8f94d27a3c0a5d64647c834 P 6344097e897541a8aa6ffd7260f93abb [term 1 FOLLOWER]: Refusing update from remote peer 48863d8e86554237b2554a088d69c8bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:33.702023 4979 consensus_queue.cc:1035] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6344097e897541a8aa6ffd7260f93abb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:33.702214 4975 consensus_queue.cc:1035] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f49ecbb1614d819841ccecd83114d6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:33.709770 4982 mvcc.cc:205] Tried to move back new op lower bound from 7332798028592439296 to 7332798028169691136. Current Snapshot: MvccSnapshot[applied={T|T < 7332798028592439296}]
I20260924 07:11:33.710155 4984 mvcc.cc:205] Tried to move back new op lower bound from 7332798028592439296 to 7332798028169691136. Current Snapshot: MvccSnapshot[applied={T|T < 7332798028592439296}]
I20260924 07:11:33.712615 4983 mvcc.cc:205] Tried to move back new op lower bound from 7332798028592439296 to 7332798028169691136. Current Snapshot: MvccSnapshot[applied={T|T < 7332798028592439296}]
I20260924 07:11:33.719822 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:33.720011 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:33.722349 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44f51cfa68d544c389f17cbf2d3ba5f4: Sending DeleteTablet for 3 replicas of tablet f7d41c1fc8f94d27a3c0a5d64647c834
I20260924 07:11:33.723188 4753 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32792
I20260924 07:11:33.723261 4620 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49548
I20260924 07:11:33.723769 4992 tablet_replica.cc:330] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: stopping tablet replica
I20260924 07:11:33.723955 4886 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d41c1fc8f94d27a3c0a5d64647c834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34028
I20260924 07:11:33.724151 4992 raft_consensus.cc:2229] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:33.724493 4992 raft_consensus.cc:2258] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:33.724648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4558
I20260924 07:11:33.726967 4992 ts_tablet_manager.cc:1905] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:33.729781 4992 ts_tablet_manager.cc:1918] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:33.729931 4992 log.cc:1196] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/f7d41c1fc8f94d27a3c0a5d64647c834
I20260924 07:11:33.730201 4994 tablet_replica.cc:330] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: stopping tablet replica
I20260924 07:11:33.730261 4992 ts_tablet_manager.cc:1939] T f7d41c1fc8f94d27a3c0a5d64647c834 P 25f49ecbb1614d819841ccecd83114d6: Deleting consensus metadata
I20260924 07:11:33.730598 4994 raft_consensus.cc:2229] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:33.731074 4994 raft_consensus.cc:2258] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:33.731743 4482 catalog_manager.cc:4841] TS 25f49ecbb1614d819841ccecd83114d6 (127.0.98.2:38121): tablet f7d41c1fc8f94d27a3c0a5d64647c834 (table test-table [id=bc4097aa80274ceea4aa1dbf752f3fe3]) successfully deleted
W20260924 07:11:33.732081 4480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35473 (error 108)
I20260924 07:11:33.732745 4994 ts_tablet_manager.cc:1905] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:33.732986 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.732718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35473 (local address 127.0.0.1:49558)
0924 07:11:33.732838 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:33.732905 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:11:33.733078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4691
W20260924 07:11:33.733242 4480 catalog_manager.cc:4584] TS 6344097e897541a8aa6ffd7260f93abb (127.0.98.1:35473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7d41c1fc8f94d27a3c0a5d64647c834: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35473: connect: Connection refused (error 111)
I20260924 07:11:33.735452 4994 ts_tablet_manager.cc:1918] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:33.735554 4994 log.cc:1196] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/f7d41c1fc8f94d27a3c0a5d64647c834
I20260924 07:11:33.735852 4994 ts_tablet_manager.cc:1939] T f7d41c1fc8f94d27a3c0a5d64647c834 P 48863d8e86554237b2554a088d69c8bf: Deleting consensus metadata
I20260924 07:11:33.737160 4480 catalog_manager.cc:4841] TS 48863d8e86554237b2554a088d69c8bf (127.0.98.3:37547): tablet f7d41c1fc8f94d27a3c0a5d64647c834 (table test-table [id=bc4097aa80274ceea4aa1dbf752f3fe3]) successfully deleted
I20260924 07:11:33.739154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4824
I20260924 07:11:33.743961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4465
2026-09-24T07:11:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2564 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T07:11:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:33Z Disabled control of system clock
I20260924 07:11:33.771998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38047
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:11:33.882110 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:33.882427 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:33.882473 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:33.886669 4999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:33.886740 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:33.886755 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:33.886771 4999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:33.886786 4999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:33.891669 4999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:33.892799 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:33.894184 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:33.902750 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:33.902762 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:33.902750 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:33.902997 4999 server_base.cc:1034] running on GCE node
I20260924 07:11:33.903723 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:33.904882 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:33.906062 4999 hybrid_clock.cc:648] HybridClock initialized: now 1790233893906038 us; error 40 us; skew 500 ppm
I20260924 07:11:33.911787 4999 webserver.cc:460] Webserver started at http://127.0.98.62:39767/ using document root <none> and password file <none>
I20260924 07:11:33.912456 4999 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:33.912539 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:33.912743 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:33.914916 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "cd2240887eca4042bfc6af7f5a33f3ef"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "09b3b6709d8d2bd62e0d5f6cb0ec3493"
server_key_iv: "2bc6f3f82eaa9b8b065c232fa95da9db"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.915508 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd2240887eca4042bfc6af7f5a33f3ef"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "09b3b6709d8d2bd62e0d5f6cb0ec3493"
server_key_iv: "2bc6f3f82eaa9b8b065c232fa95da9db"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.919263 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:11:33.921918 5013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:33.923353 4999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:33.923539 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "cd2240887eca4042bfc6af7f5a33f3ef"
format_stamp: "Formatted at 2026-09-24 07:11:33 on dist-test-slave-cnv3"
server_key: "09b3b6709d8d2bd62e0d5f6cb0ec3493"
server_key_iv: "2bc6f3f82eaa9b8b065c232fa95da9db"
server_key_version: "encryptionkey@0"
I20260924 07:11:33.923650 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:33.928524 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:33.929163 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:33.929343 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:33.935912 4999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38047
I20260924 07:11:33.935935 5065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38047 every 8 connection(s)
I20260924 07:11:33.936865 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:33.937240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 4999
I20260924 07:11:33.937356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:33.937644 392 external_mini_cluster.cc:1428] Setting key 23999c5ab7a701fc042775469ac61eb9
I20260924 07:11:33.941072 5066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:33.943769 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:33.939071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.0.98.62:38047)
0924 07:11:33.940151 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:33.940167 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:33.940204 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:11:33.940903 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:33.940923 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:33.941324 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:33.941669 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:33.942462 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:33.942466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:33.942479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:33.942489 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:33.942543 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:33.942559 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:33.942690 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:33.942761 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:11:33.943022 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":564,"threads_started":1}
I20260924 07:11:33.947192 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: Bootstrap starting.
I20260924 07:11:33.949517 5066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:33.950250 5066 log.cc:824] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:33.952690 5066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: No bootstrap required, opened a new log
I20260924 07:11:33.956338 5066 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } }
I20260924 07:11:33.956605 5066 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:33.956642 5066 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2240887eca4042bfc6af7f5a33f3ef, State: Initialized, Role: FOLLOWER
I20260924 07:11:33.957031 5066 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } }
I20260924 07:11:33.957145 5066 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:33.957185 5066 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:33.957257 5066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:33.958181 5066 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } }
I20260924 07:11:33.958544 5066 leader_election.cc:302] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2240887eca4042bfc6af7f5a33f3ef; no voters:
I20260924 07:11:33.958832 5066 leader_election.cc:288] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:33.958983 5071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:33.959193 5071 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [term 1 LEADER]: Becoming Leader. State: Replica: cd2240887eca4042bfc6af7f5a33f3ef, State: Running, Role: LEADER
I20260924 07:11:33.959597 5071 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } }
I20260924 07:11:33.960212 5066 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:33.961238 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2240887eca4042bfc6af7f5a33f3ef. Latest consensus state: current_term: 1 leader_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } } }
I20260924 07:11:33.961360 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:33.961632 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2240887eca4042bfc6af7f5a33f3ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38047 } } }
I20260924 07:11:33.961719 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:33.961730 5080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:33.964671 5080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:33.970587 5080 catalog_manager.cc:1347] Generated new cluster ID: ebc1b2934b3946c9af2389407b3a101b
I20260924 07:11:33.970677 5080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:33.978497 5080 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:33.978873 5080 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:33.985586 5080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: Generated new TSK 0
I20260924 07:11:33.986271 5080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:11:33.990305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:11:34.103300 5090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:34.103654 5090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:34.103696 5090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:34.107918 5090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:34.107983 5090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:34.108044 5090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:34.112958 5090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:34.114048 5090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:34.115563 5090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:34.125231 5096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.125305 5095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.125252 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:34.125491 5090 server_base.cc:1034] running on GCE node
I20260924 07:11:34.126014 5090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:34.126725 5090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:34.127912 5090 hybrid_clock.cc:648] HybridClock initialized: now 1790233894127896 us; error 29 us; skew 500 ppm
I20260924 07:11:34.129881 5090 webserver.cc:460] Webserver started at http://127.0.98.1:42747/ using document root <none> and password file <none>
I20260924 07:11:34.130486 5090 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:34.130570 5090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:34.130769 5090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:34.132902 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "5fa5bebcf85f4d9f8c12ec55047619ef"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "00a19249dd28d52881254141136ab9d7"
server_key_iv: "1d12c1c39e9e3d69c6fb0f00a1770219"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.133445 5090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa5bebcf85f4d9f8c12ec55047619ef"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "00a19249dd28d52881254141136ab9d7"
server_key_iv: "1d12c1c39e9e3d69c6fb0f00a1770219"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.137276 5090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:34.139794 5104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.141158 5090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:34.141287 5090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "5fa5bebcf85f4d9f8c12ec55047619ef"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "00a19249dd28d52881254141136ab9d7"
server_key_iv: "1d12c1c39e9e3d69c6fb0f00a1770219"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.141404 5090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:34.146077 5090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:34.146677 5090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:34.146833 5090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:34.147361 5090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:34.148279 5090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:34.148326 5090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.148386 5090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:34.148413 5090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.156776 5090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33637
I20260924 07:11:34.156826 5217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33637 every 8 connection(s)
I20260924 07:11:34.157711 5090 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:34.164072 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.159689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57413 (local address 127.0.98.62:38047)
0924 07:11:34.159850 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.159853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.160895 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 07:11:34.162334 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.162338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.162384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.162488 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.163632 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.163635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.163637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.163648 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.163667 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.163674 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.163757 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.163884 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.163929 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:11:34.165144 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.160002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38047 (local address 127.0.98.1:57413)
0924 07:11:34.160742 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.160790 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.162114 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.162651 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.162658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.162748 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.163314 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.163316 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.163357 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.163483 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.163747 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.163755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.163761 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.163946 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.164207 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260924 07:11:34.166432 5218 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38047
I20260924 07:11:34.166692 5218 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:34.166693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5090
I20260924 07:11:34.166788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:34.167115 392 external_mini_cluster.cc:1428] Setting key 2a8bb863f702ff02ab0f6b6b394093fd
I20260924 07:11:34.167176 5218 heartbeater.cc:511] Master 127.0.98.62:38047 requested a full tablet report, sending...
I20260924 07:11:34.168803 5029 ts_manager.cc:194] Registered new tserver with Master: 5fa5bebcf85f4d9f8c12ec55047619ef (127.0.98.1:33637)
I20260924 07:11:34.169126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:11:34.279906 5221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:34.280193 5221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:34.280233 5221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:34.284349 5221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:34.284415 5221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:34.284484 5221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:34.289403 5221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:34.290488 5221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:34.291908 5221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:34.301628 5226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:34.301692 5221 server_base.cc:1034] running on GCE node
W20260924 07:11:34.301640 5227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.301628 5229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:34.302282 5221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:34.302923 5221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:34.304100 5221 hybrid_clock.cc:648] HybridClock initialized: now 1790233894304071 us; error 41 us; skew 500 ppm
I20260924 07:11:34.305970 5221 webserver.cc:460] Webserver started at http://127.0.98.2:38261/ using document root <none> and password file <none>
I20260924 07:11:34.306520 5221 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:34.306581 5221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:34.306766 5221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:34.308806 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae75c45dc2334a08845edd75ac222b72"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "438daccd53ea987feee3f1f32363f597"
server_key_iv: "69aa949efc140a9ec60bde487b7c40dc"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.309343 5221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae75c45dc2334a08845edd75ac222b72"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "438daccd53ea987feee3f1f32363f597"
server_key_iv: "69aa949efc140a9ec60bde487b7c40dc"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.312939 5221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:11:34.315331 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.316623 5221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:34.316736 5221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "ae75c45dc2334a08845edd75ac222b72"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "438daccd53ea987feee3f1f32363f597"
server_key_iv: "69aa949efc140a9ec60bde487b7c40dc"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.316841 5221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:34.321450 5221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:34.322093 5221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:34.322228 5221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:34.322757 5221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:34.323694 5221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:34.323741 5221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.323779 5221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:34.323799 5221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.332466 5348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46297 every 8 connection(s)
I20260924 07:11:34.332444 5221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46297
I20260924 07:11:34.333616 5221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:34.335659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5221
I20260924 07:11:34.335793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:34.336083 392 external_mini_cluster.cc:1428] Setting key 69a786e779c0b255c4c9dbd90949dfbd
I20260924 07:11:34.338480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--builtin_ntp_servers=127.0.98.20:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 07:11:34.340603 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.335599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51783 (local address 127.0.98.62:38047)
0924 07:11:34.335756 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.335761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.337089 (+ 1328us) server_negotiation.cc:407] Connection header received
0924 07:11:34.338666 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.338670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.338716 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.338821 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.340165 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.340169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.340171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.340186 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.340211 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.340221 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.340331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.340417 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.340462 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:11:34.341871 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.336134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38047 (local address 127.0.98.2:51783)
0924 07:11:34.336948 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.336992 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.338383 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.339056 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.339066 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.339196 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.339837 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.339839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.339911 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.339999 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.340304 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.340309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.340312 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.340439 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.340647 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":135,"threads_started":1}
I20260924 07:11:34.343248 5349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38047
I20260924 07:11:34.343544 5349 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:34.343968 5349 heartbeater.cc:511] Master 127.0.98.62:38047 requested a full tablet report, sending...
I20260924 07:11:34.345161 5029 ts_manager.cc:194] Registered new tserver with Master: ae75c45dc2334a08845edd75ac222b72 (127.0.98.2:46297)
W20260924 07:11:34.454391 5352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:34.454711 5352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:34.454753 5352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:34.458966 5352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:34.459048 5352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:34.459112 5352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:34.464574 5352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:34.465788 5352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:34.467208 5352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:34.476697 5360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.476692 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.476697 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:34.476883 5352 server_base.cc:1034] running on GCE node
I20260924 07:11:34.477380 5352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:34.478052 5352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:34.479238 5352 hybrid_clock.cc:648] HybridClock initialized: now 1790233894479216 us; error 33 us; skew 500 ppm
I20260924 07:11:34.481279 5352 webserver.cc:460] Webserver started at http://127.0.98.3:40545/ using document root <none> and password file <none>
I20260924 07:11:34.481840 5352 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:34.481901 5352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:34.482100 5352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:34.484273 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "bc468c4ed7b046adb4230d7a42e492fb"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "12cf482f139be9d584550af1572741f9"
server_key_iv: "f48deae3cd55467378fcfd6e71370f46"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.484807 5352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bc468c4ed7b046adb4230d7a42e492fb"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "12cf482f139be9d584550af1572741f9"
server_key_iv: "f48deae3cd55467378fcfd6e71370f46"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.488655 5352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:34.491168 5366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.492509 5352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:34.492619 5352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "bc468c4ed7b046adb4230d7a42e492fb"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "12cf482f139be9d584550af1572741f9"
server_key_iv: "f48deae3cd55467378fcfd6e71370f46"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.492739 5352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:34.497277 5352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:34.497862 5352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:34.498006 5352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:34.498518 5352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:34.499430 5352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:34.499480 5352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.499517 5352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:34.499531 5352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.507766 5352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40981
I20260924 07:11:34.507789 5479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40981 every 8 connection(s)
I20260924 07:11:34.508661 5352 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:34.514883 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.510599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40595 (local address 127.0.98.62:38047)
0924 07:11:34.510725 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.510728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.511861 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 07:11:34.513213 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.513217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.513262 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.513336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.514435 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.514438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.514440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.514451 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.514472 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.514482 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.514575 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.514702 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.514746 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:11:34.515588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5352
I20260924 07:11:34.515697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:34.515879 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.510994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38047 (local address 127.0.98.3:40595)
0924 07:11:34.511709 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.511752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.512996 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.513520 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.513527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.513619 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.514179 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.514180 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.514222 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.514290 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.514568 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.514576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.514582 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.514739 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.514978 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 07:11:34.515950 392 external_mini_cluster.cc:1428] Setting key 38e5620539b1c3ffae7f20db7d0d6bd3
I20260924 07:11:34.517122 5480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38047
I20260924 07:11:34.517354 5480 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:34.517755 5480 heartbeater.cc:511] Master 127.0.98.62:38047 requested a full tablet report, sending...
I20260924 07:11:34.518738 5030 ts_manager.cc:194] Registered new tserver with Master: bc468c4ed7b046adb4230d7a42e492fb (127.0.98.3:40981)
I20260924 07:11:34.530092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:11:34.533803 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.532019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.0.98.62:38047)
0924 07:11:34.532176 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.532179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.532288 (+ 109us) server_negotiation.cc:407] Connection header received
0924 07:11:34.532437 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.532440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.532477 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.532569 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.533343 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.533347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.533349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.533361 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.533380 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.533388 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.533470 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.533610 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.533648 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:34.536891 5030 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:34.540094 5030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:34.552500 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.549188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40981 (local address 127.0.0.1:33112)
0924 07:11:34.549935 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.549947 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.550010 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.550835 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.550839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.550855 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.551285 (+ 430us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.551287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.551298 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.551325 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.552114 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.552124 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.552130 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.552240 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.552309 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":112,"threads_started":1}
I20260924 07:11:34.552588 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.549261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.2:46297)
0924 07:11:34.549984 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.549996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.550020 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:11:34.550088 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.550094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.550310 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.551085 (+ 775us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.552002 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.552006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.552014 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.552022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.552062 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.552084 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.552221 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.552366 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.552439 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":158,"threads_started":1}
I20260924 07:11:34.552525 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.549120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46297 (local address 127.0.0.1:59120)
0924 07:11:34.549575 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.549631 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.549739 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.550487 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.550493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.550576 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.551168 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.551170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.551194 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.551269 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.552238 (+ 969us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.552241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.552245 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.552330 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.552365 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":81,"threads_started":1}
I20260924 07:11:34.553050 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.549974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.0.98.3:40981)
0924 07:11:34.550394 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.550405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.550428 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:11:34.550481 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.550486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.550682 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.551074 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.551926 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.551932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.551944 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.551954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.551992 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.552017 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.552126 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.552314 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.552389 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":125,"threads_started":1}
I20260924 07:11:34.553069 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.549899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33637 (local address 127.0.0.1:60136)
0924 07:11:34.550670 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.550684 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.550755 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.551308 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.551311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.551327 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.551775 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.551777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.551787 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.551812 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.552854 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.552858 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.552862 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.552933 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.552964 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":87,"threads_started":1}
I20260924 07:11:34.553440 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.550285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60136 (local address 127.0.98.1:33637)
0924 07:11:34.550837 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.550853 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.550875 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:11:34.550921 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.550926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.551114 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.551235 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.551926 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.551932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.551944 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.551954 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.551993 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.552027 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.552126 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.553218 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.553287 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":139,"threads_started":1}
I20260924 07:11:34.555856 5414 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 (DEFAULT_TABLE table=test-table [id=724e670ad9c446b8add7bccf859b6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:34.555975 5283 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 (DEFAULT_TABLE table=test-table [id=724e670ad9c446b8add7bccf859b6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:34.555974 5152 tablet_service.cc:1467] Processing CreateTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 (DEFAULT_TABLE table=test-table [id=724e670ad9c446b8add7bccf859b6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:34.556962 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ac6cd46934fc28dd101b5867ef994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:34.557000 5283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ac6cd46934fc28dd101b5867ef994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:34.557000 5152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9ac6cd46934fc28dd101b5867ef994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:34.563319 5497 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Bootstrap starting.
I20260924 07:11:34.564481 5498 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Bootstrap starting.
I20260924 07:11:34.564986 5499 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Bootstrap starting.
I20260924 07:11:34.565832 5497 tablet_bootstrap.cc:654] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:34.566614 5497 log.cc:824] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:34.566939 5499 tablet_bootstrap.cc:654] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:34.567735 5499 log.cc:824] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:34.568758 5497 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: No bootstrap required, opened a new log
I20260924 07:11:34.568951 5498 tablet_bootstrap.cc:654] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:34.568962 5497 ts_tablet_manager.cc:1397] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:34.569375 5499 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: No bootstrap required, opened a new log
I20260924 07:11:34.569558 5499 ts_tablet_manager.cc:1397] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:11:34.569741 5498 log.cc:824] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:34.571861 5498 tablet_bootstrap.cc:492] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: No bootstrap required, opened a new log
I20260924 07:11:34.572119 5498 ts_tablet_manager.cc:1397] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:11:34.573230 5499 raft_consensus.cc:348] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.573450 5499 raft_consensus.cc:374] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:34.573273 5497 raft_consensus.cc:348] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.573521 5497 raft_consensus.cc:374] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:34.573539 5499 raft_consensus.cc:729] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa5bebcf85f4d9f8c12ec55047619ef, State: Initialized, Role: FOLLOWER
I20260924 07:11:34.573606 5497 raft_consensus.cc:729] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc468c4ed7b046adb4230d7a42e492fb, State: Initialized, Role: FOLLOWER
I20260924 07:11:34.573995 5499 consensus_queue.cc:260] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.574028 5497 consensus_queue.cc:260] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.574666 5218 heartbeater.cc:503] Master 127.0.98.62:38047 was elected leader, sending a full tablet report...
I20260924 07:11:34.575096 5497 ts_tablet_manager.cc:1428] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:11:34.575469 5480 heartbeater.cc:503] Master 127.0.98.62:38047 was elected leader, sending a full tablet report...
I20260924 07:11:34.576043 5499 ts_tablet_manager.cc:1428] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:11:34.576344 5498 raft_consensus.cc:348] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.576617 5498 raft_consensus.cc:374] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:34.576707 5498 raft_consensus.cc:729] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae75c45dc2334a08845edd75ac222b72, State: Initialized, Role: FOLLOWER
I20260924 07:11:34.577109 5498 consensus_queue.cc:260] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.577945 5349 heartbeater.cc:503] Master 127.0.98.62:38047 was elected leader, sending a full tablet report...
I20260924 07:11:34.578060 5498 ts_tablet_manager.cc:1428] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 07:11:34.584544 5350 tablet.cc:2369] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:34.655652 5503 raft_consensus.cc:482] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:34.655875 5503 raft_consensus.cc:504] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.657125 5503 leader_election.cc:288] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae75c45dc2334a08845edd75ac222b72 (127.0.98.2:46297), bc468c4ed7b046adb4230d7a42e492fb (127.0.98.3:40981)
W20260924 07:11:34.658875 5219 tablet.cc:2369] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:34.659210 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.657425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46297 (local address 127.0.98.1:34971)
0924 07:11:34.657631 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.657646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.657736 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.658016 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.658019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.658034 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.658487 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.658488 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.658498 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.658526 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.658925 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.658928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.658931 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.659030 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.659070 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 07:11:34.659276 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.657572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34971 (local address 127.0.98.2:46297)
0924 07:11:34.657705 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.657710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.657727 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:11:34.657862 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.657864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.657901 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.657971 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.658768 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.658772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.658775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.658789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.658812 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.658822 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.658942 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.659111 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.659158 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:11:34.659310 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.657663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59115 (local address 127.0.98.3:40981)
0924 07:11:34.657782 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.657787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.658046 (+ 259us) server_negotiation.cc:407] Connection header received
0924 07:11:34.658101 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.658104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.658142 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.658214 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.658901 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.658905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.658907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.658922 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.658943 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.658954 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.659032 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.659163 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.659201 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:34.659658 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.657549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40981 (local address 127.0.98.1:59115)
0924 07:11:34.657928 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.657939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:34.658030 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:34.658248 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:34.658250 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:34.658262 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:34.658696 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:11:34.658698 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:34.658711 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:11:34.658735 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:34.659067 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:34.659070 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:34.659073 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:11:34.659463 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 07:11:34.659505 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":18,"thread_start_us":75,"threads_started":1}
I20260924 07:11:34.660104 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ac6cd46934fc28dd101b5867ef994" candidate_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae75c45dc2334a08845edd75ac222b72" is_pre_election: true
I20260924 07:11:34.660225 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ac6cd46934fc28dd101b5867ef994" candidate_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc468c4ed7b046adb4230d7a42e492fb" is_pre_election: true
I20260924 07:11:34.660422 5303 raft_consensus.cc:2454] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa5bebcf85f4d9f8c12ec55047619ef in term 0.
I20260924 07:11:34.660496 5434 raft_consensus.cc:2454] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa5bebcf85f4d9f8c12ec55047619ef in term 0.
I20260924 07:11:34.660929 5107 leader_election.cc:302] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa5bebcf85f4d9f8c12ec55047619ef, ae75c45dc2334a08845edd75ac222b72; no voters:
I20260924 07:11:34.661195 5503 raft_consensus.cc:2781] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:34.661274 5503 raft_consensus.cc:482] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:34.661305 5503 raft_consensus.cc:3037] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:34.662407 5503 raft_consensus.cc:504] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.662745 5503 leader_election.cc:288] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [CANDIDATE]: Term 1 election: Requested vote from peers ae75c45dc2334a08845edd75ac222b72 (127.0.98.2:46297), bc468c4ed7b046adb4230d7a42e492fb (127.0.98.3:40981)
I20260924 07:11:34.663085 5303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ac6cd46934fc28dd101b5867ef994" candidate_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae75c45dc2334a08845edd75ac222b72"
I20260924 07:11:34.663085 5434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab9ac6cd46934fc28dd101b5867ef994" candidate_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc468c4ed7b046adb4230d7a42e492fb"
I20260924 07:11:34.663205 5434 raft_consensus.cc:3037] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:34.663206 5303 raft_consensus.cc:3037] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:34.664230 5434 raft_consensus.cc:2454] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa5bebcf85f4d9f8c12ec55047619ef in term 1.
I20260924 07:11:34.664283 5303 raft_consensus.cc:2454] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa5bebcf85f4d9f8c12ec55047619ef in term 1.
I20260924 07:11:34.664505 5106 leader_election.cc:302] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa5bebcf85f4d9f8c12ec55047619ef, bc468c4ed7b046adb4230d7a42e492fb; no voters:
I20260924 07:11:34.664657 5503 raft_consensus.cc:2781] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:34.664882 5503 raft_consensus.cc:686] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 1 LEADER]: Becoming Leader. State: Replica: 5fa5bebcf85f4d9f8c12ec55047619ef, State: Running, Role: LEADER
I20260924 07:11:34.665225 5503 consensus_queue.cc:237] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } }
I20260924 07:11:34.668392 5024 catalog_manager.cc:5495] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fa5bebcf85f4d9f8c12ec55047619ef (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fa5bebcf85f4d9f8c12ec55047619ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33637 } health_report { overall_health: HEALTHY } } }
I20260924 07:11:34.706743 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.705279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.98.1:33637)
0924 07:11:34.705482 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.705485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.705498 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:34.705576 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.705578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.705635 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.705715 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.706344 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.706347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.706349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.706360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.706380 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.706388 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.706466 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.706565 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.706599 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:34.712798 5303 raft_consensus.cc:1261] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 1 FOLLOWER]: Refusing update from remote peer 5fa5bebcf85f4d9f8c12ec55047619ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:34.712792 5434 raft_consensus.cc:1261] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 1 FOLLOWER]: Refusing update from remote peer 5fa5bebcf85f4d9f8c12ec55047619ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:34.713524 5507 consensus_queue.cc:1035] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc468c4ed7b046adb4230d7a42e492fb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:34.713712 5503 consensus_queue.cc:1035] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae75c45dc2334a08845edd75ac222b72" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:34.721481 5510 mvcc.cc:205] Tried to move back new op lower bound from 7332798032736145408 to 7332798032551485440. Current Snapshot: MvccSnapshot[applied={T|T < 7332798032736145408}]
I20260924 07:11:34.725042 5512 mvcc.cc:205] Tried to move back new op lower bound from 7332798032736145408 to 7332798032551485440. Current Snapshot: MvccSnapshot[applied={T|T < 7332798032736145408}]
I20260924 07:11:34.732815 5030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:34.733019 5030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:34.735809 5030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd2240887eca4042bfc6af7f5a33f3ef: Sending DeleteTablet for 3 replicas of tablet ab9ac6cd46934fc28dd101b5867ef994
I20260924 07:11:34.736742 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:34 UTC) from {username='slave'} at 127.0.0.1:60136
I20260924 07:11:34.736773 5283 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:34 UTC) from {username='slave'} at 127.0.0.1:59120
I20260924 07:11:34.736833 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet ab9ac6cd46934fc28dd101b5867ef994 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:34 UTC) from {username='slave'} at 127.0.0.1:33112
I20260924 07:11:34.737345 5523 tablet_replica.cc:330] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: stopping tablet replica
I20260924 07:11:34.737402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5090
I20260924 07:11:34.737471 5522 tablet_replica.cc:330] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef: stopping tablet replica
I20260924 07:11:34.737800 5522 raft_consensus.cc:2229] T ab9ac6cd46934fc28dd101b5867ef994 P 5fa5bebcf85f4d9f8c12ec55047619ef [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:34.738065 5521 tablet_replica.cc:330] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: stopping tablet replica
I20260924 07:11:34.738337 5521 raft_consensus.cc:2229] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:34.738348 5523 raft_consensus.cc:2229] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:34.738626 5523 raft_consensus.cc:2258] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:34.738636 5521 raft_consensus.cc:2258] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:34.740020 5523 ts_tablet_manager.cc:1905] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:34.741642 5521 ts_tablet_manager.cc:1905] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:34.742448 5523 ts_tablet_manager.cc:1918] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:34.742539 5523 log.cc:1196] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/ab9ac6cd46934fc28dd101b5867ef994
I20260924 07:11:34.742797 5523 ts_tablet_manager.cc:1939] T ab9ac6cd46934fc28dd101b5867ef994 P bc468c4ed7b046adb4230d7a42e492fb: Deleting consensus metadata
I20260924 07:11:34.744277 5521 ts_tablet_manager.cc:1918] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:34.744374 5521 log.cc:1196] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/ab9ac6cd46934fc28dd101b5867ef994
I20260924 07:11:34.744339 5015 catalog_manager.cc:4841] TS bc468c4ed7b046adb4230d7a42e492fb (127.0.98.3:40981): tablet ab9ac6cd46934fc28dd101b5867ef994 (table test-table [id=724e670ad9c446b8add7bccf859b6e93]) successfully deleted
I20260924 07:11:34.744683 5521 ts_tablet_manager.cc:1939] T ab9ac6cd46934fc28dd101b5867ef994 P ae75c45dc2334a08845edd75ac222b72: Deleting consensus metadata
W20260924 07:11:34.745287 5015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33637 (error 108)
I20260924 07:11:34.745492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5221
I20260924 07:11:34.747318 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.746984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33637 (local address 127.0.0.1:60142)
0924 07:11:34.747149 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.747229 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 07:11:34.747494 5015 catalog_manager.cc:4584] TS 5fa5bebcf85f4d9f8c12ec55047619ef (127.0.98.1:33637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab9ac6cd46934fc28dd101b5867ef994: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33637: connect: Connection refused (error 111)
I20260924 07:11:34.750871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5352
I20260924 07:11:34.751287 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.750987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46297 (local address 127.0.0.1:59130)
0924 07:11:34.751158 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:34.751207 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 07:11:34.756259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 4999
2026-09-24T07:11:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1011 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 07:11:34.776376 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:11:34.777948 392 webserver.cc:460] Webserver started at https://127.0.0.1:37051/ using document root <none> and password file <none>
I20260924 07:11:34.778116 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:11:34.778368 392 webserver.cc:460] Webserver started at http://127.0.0.1:39029/ using document root <none> and password file <none>
2026-09-24T07:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:34Z Disabled control of system clock
I20260924 07:11:34.786921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39259
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39259
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37051/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/testchainca.cert with env {}
W20260924 07:11:34.898654 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:34.898944 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:34.898984 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:34.903270 5530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:34.903349 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:34.903364 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:34.903401 5530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:34.903416 5530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:11:34.903431 5530 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:11:34.903445 5530 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37051/jwks.json
I20260924 07:11:34.909657 5530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39259
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37051/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:34.910794 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:34.911923 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:34.920341 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.920344 5535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:34.920341 5536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:34.920634 5530 server_base.cc:1034] running on GCE node
I20260924 07:11:34.921272 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:34.922461 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:34.923635 5530 hybrid_clock.cc:648] HybridClock initialized: now 1790233894923615 us; error 38 us; skew 500 ppm
I20260924 07:11:34.925798 5530 webserver.cc:460] Webserver started at http://127.0.98.62:36071/ using document root <none> and password file <none>
I20260924 07:11:34.926343 5530 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:34.926414 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:34.926607 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:34.928865 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "9a4d181449064907a1cd5d12e70fdcae"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "17ebf48fa2fe121c48192da697c263f4"
server_key_iv: "e4c05d074396b602bad5038416baaf7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.929533 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a4d181449064907a1cd5d12e70fdcae"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "17ebf48fa2fe121c48192da697c263f4"
server_key_iv: "e4c05d074396b602bad5038416baaf7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.933310 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:34.935886 5544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:34.937202 5530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:34.937331 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "9a4d181449064907a1cd5d12e70fdcae"
format_stamp: "Formatted at 2026-09-24 07:11:34 on dist-test-slave-cnv3"
server_key: "17ebf48fa2fe121c48192da697c263f4"
server_key_iv: "e4c05d074396b602bad5038416baaf7e"
server_key_version: "encryptionkey@0"
I20260924 07:11:34.937440 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:34.966326 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:34.967062 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:34.967260 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:34.973982 5530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39259
I20260924 07:11:34.974042 5596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39259 every 8 connection(s)
I20260924 07:11:34.974939 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:34.978153 5597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:34.982929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5530
I20260924 07:11:34.983060 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:34.983333 392 external_mini_cluster.cc:1428] Setting key 3dc1dea588d438366233078cbde849de
I20260924 07:11:34.984050 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae: Bootstrap starting.
I20260924 07:11:34.986434 5597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:34.987166 5597 log.cc:824] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:34.988973 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae: No bootstrap required, opened a new log
I20260924 07:11:34.992476 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:34.984731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.0.98.62:39259)
0924 07:11:34.985369 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:34.985382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:34.985418 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:34.986108 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:34.986126 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:34.986558 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:34.986882 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:34.987335 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:34.988383 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:34.989294 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:34.989768 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:34.989923 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:34.990692 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:34.990697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:34.990710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:34.991100 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:34.991156 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:34.991176 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:34.991262 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:34.991452 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:11:34.991719 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":127,"threads_started":1}
I20260924 07:11:34.992564 5597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260924 07:11:34.992863 5597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:34.992930 5597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a4d181449064907a1cd5d12e70fdcae, State: Initialized, Role: FOLLOWER
I20260924 07:11:34.993418 5597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260924 07:11:34.993578 5597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:34.993633 5597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:34.993721 5597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:34.994769 5597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260924 07:11:34.995108 5597 leader_election.cc:302] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a4d181449064907a1cd5d12e70fdcae; no voters:
I20260924 07:11:34.995428 5597 leader_election.cc:288] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:34.995519 5602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:34.995553 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:11:34.995707 5602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [term 1 LEADER]: Becoming Leader. State: Replica: 9a4d181449064907a1cd5d12e70fdcae, State: Running, Role: LEADER
I20260924 07:11:34.996033 5602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } }
I20260924 07:11:34.996891 5597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:34.997802 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a4d181449064907a1cd5d12e70fdcae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } } }
I20260924 07:11:34.997941 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:34.998297 5611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:34.998253 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a4d181449064907a1cd5d12e70fdcae. Latest consensus state: current_term: 1 leader_uuid: "9a4d181449064907a1cd5d12e70fdcae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a4d181449064907a1cd5d12e70fdcae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39259 } } }
I20260924 07:11:34.998493 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.001993 5611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:35.007547 5611 catalog_manager.cc:1347] Generated new cluster ID: e7956f427d454e04a8583b1a7ed19de0
I20260924 07:11:35.007608 5611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:35.016893 5611 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:35.017946 5611 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:35.024222 5611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a4d181449064907a1cd5d12e70fdcae: Generated new TSK 0
I20260924 07:11:35.025164 5611 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:11:35.068034 5627 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 07:11:35.068151 5627 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:11:35.068630 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.044149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.0.98.62:39259)
0924 07:11:35.044279 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.044284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.044484 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:11:35.044705 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.044707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.044757 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.044841 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:11:35.045188 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.045764 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.046993 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.047221 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.047297 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.047748 (+ 451us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:11:35.068231 (+ 20483us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:11:35.068497 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 07:11:35.068719 5600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 07:11:35.070075 5626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:35.044015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39259 (local address 127.0.0.1:40990)
0924 07:11:35.044385 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:35.044401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:35.044552 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:35.044883 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:35.044886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:35.044894 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:11:35.045034 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.045042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.045888 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.045892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.046873 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.046884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.047314 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.047336 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.047457 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.047519 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:11:35.068486 (+ 20967us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:11:35.068753 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39259: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:39259: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 07:11:35.070935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5530
2026-09-24T07:11:35Z chronyd exiting
I20260924 07:11:35.107544 392 test_util.cc:175] -----------------------------------------------
I20260924 07:11:35.107632 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233866113376-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 07:11:35.108659 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:11:35.109498 392 webserver.cc:460] Webserver started at https://127.0.0.1:41995/ using document root <none> and password file <none>
I20260924 07:11:35.109582 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:11:35.109833 392 webserver.cc:460] Webserver started at http://127.0.0.1:32981/ using document root <none> and password file <none>
2026-09-24T07:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:35Z Disabled control of system clock
I20260924 07:11:35.117949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41561
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41995/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/testchainca.cert with env {}
W20260924 07:11:35.225864 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:35.226159 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:35.226212 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:35.230268 5633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:35.230337 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:35.230358 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:35.230379 5633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:35.230398 5633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:11:35.230422 5633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:11:35.230448 5633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41995/jwks.json
I20260924 07:11:35.235913 5633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41995/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:35.236982 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:35.237983 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:35.246101 5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:35.246117 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:35.246106 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:35.246330 5633 server_base.cc:1034] running on GCE node
I20260924 07:11:35.246950 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:35.248064 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:35.249254 5633 hybrid_clock.cc:648] HybridClock initialized: now 1790233895249227 us; error 36 us; skew 500 ppm
I20260924 07:11:35.251252 5633 webserver.cc:460] Webserver started at http://127.0.98.62:37903/ using document root <none> and password file <none>
I20260924 07:11:35.251847 5633 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:35.251915 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:35.252102 5633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:35.254225 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "729b03bcd08e4ac1a36cfe8a32b11657"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "5192982e0f304f2d41bede5a6b002f22"
server_key_iv: "b20e38795394bcd4f6719c32cc991e8e"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.254802 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "729b03bcd08e4ac1a36cfe8a32b11657"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "5192982e0f304f2d41bede5a6b002f22"
server_key_iv: "b20e38795394bcd4f6719c32cc991e8e"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.258571 5633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:35.261205 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:35.262495 5633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:35.262614 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "729b03bcd08e4ac1a36cfe8a32b11657"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "5192982e0f304f2d41bede5a6b002f22"
server_key_iv: "b20e38795394bcd4f6719c32cc991e8e"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.262724 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:35.277571 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:35.278293 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:35.278469 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:35.285368 5633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41561
I20260924 07:11:35.285401 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41561 every 8 connection(s)
I20260924 07:11:35.286293 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:35.289454 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:35.293599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5633
I20260924 07:11:35.293715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:35.293951 392 external_mini_cluster.cc:1428] Setting key 7bb8b204251a65076b94f470412a0508
I20260924 07:11:35.295251 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657: Bootstrap starting.
I20260924 07:11:35.297581 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:35.298301 5700 log.cc:824] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:35.300263 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657: No bootstrap required, opened a new log
I20260924 07:11:35.303020 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.295315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.98.62:41561)
0924 07:11:35.295992 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.296007 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.296043 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:35.296699 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.296715 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.297156 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.297470 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:35.297904 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.299009 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.299925 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.300417 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.300565 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.301334 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:35.301340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:35.301354 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:35.301738 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:35.301790 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:35.301810 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:35.301894 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:35.302036 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:11:35.302293 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260924 07:11:35.303913 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } }
I20260924 07:11:35.304184 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:35.304255 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729b03bcd08e4ac1a36cfe8a32b11657, State: Initialized, Role: FOLLOWER
I20260924 07:11:35.304668 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } }
I20260924 07:11:35.304795 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:35.304860 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:35.304948 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:35.305876 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } }
I20260924 07:11:35.305951 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:11:35.306241 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729b03bcd08e4ac1a36cfe8a32b11657; no voters:
I20260924 07:11:35.306574 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:35.306689 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:35.306888 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [term 1 LEADER]: Becoming Leader. State: Replica: 729b03bcd08e4ac1a36cfe8a32b11657, State: Running, Role: LEADER
I20260924 07:11:35.307209 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } }
I20260924 07:11:35.307875 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:35.308380 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729b03bcd08e4ac1a36cfe8a32b11657. Latest consensus state: current_term: 1 leader_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } } }
I20260924 07:11:35.308514 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.308802 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729b03bcd08e4ac1a36cfe8a32b11657" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41561 } } }
I20260924 07:11:35.308889 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.308929 5711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:35.311753 5711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:35.317618 5711 catalog_manager.cc:1347] Generated new cluster ID: 98a265507b304e079fea385628b928ca
I20260924 07:11:35.317701 5711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:35.326792 5711 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:35.327783 5711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:35.331913 5711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729b03bcd08e4ac1a36cfe8a32b11657: Generated new TSK 0
I20260924 07:11:35.332614 5711 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:11:35.362841 5730 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 07:11:35.362926 5730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:11:35.363763 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.347707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.0.98.62:41561)
0924 07:11:35.347830 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.347835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.347918 (+ 83us) server_negotiation.cc:407] Connection header received
0924 07:11:35.348056 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.348059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.348118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.348237 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:11:35.348472 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.349312 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.350158 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.350516 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.350625 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.350900 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:11:35.363254 (+ 12354us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:11:35.363609 (+ 355us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:11:35.363826 5729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:35.347561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41561 (local address 127.0.0.1:55148)
0924 07:11:35.347835 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:35.347848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:35.347942 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:35.348266 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:35.348269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:35.348277 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:11:35.348371 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.348376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.349419 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.349422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.350046 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.350053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.350623 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.350641 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.350705 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.350728 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:11:35.363514 (+ 12786us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:11:35.363683 (+ 169us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41561: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
W20260924 07:11:35.363831 5703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55148: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 07:11:35.364603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5633
2026-09-24T07:11:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (294 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 07:11:35.402805 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:11:35.403738 392 webserver.cc:460] Webserver started at https://127.0.0.1:40499/ using document root <none> and password file <none>
I20260924 07:11:35.403851 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:11:35.404151 392 webserver.cc:460] Webserver started at http://127.0.0.1:41471/ using document root <none> and password file <none>
2026-09-24T07:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:35Z Disabled control of system clock
I20260924 07:11:35.414662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40499/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:11:35.523808 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:35.524078 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:35.524116 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:35.528098 5736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:35.528167 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:35.528182 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:35.528198 5736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:35.528211 5736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:11:35.528225 5736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:11:35.528237 5736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40499/jwks.json
I20260924 07:11:35.533114 5736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40499/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:35.534245 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:35.535791 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:35.544025 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:35.544142 5736 server_base.cc:1034] running on GCE node
W20260924 07:11:35.544010 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:35.544010 5744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:35.544849 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:35.545969 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:35.547142 5736 hybrid_clock.cc:648] HybridClock initialized: now 1790233895547126 us; error 29 us; skew 500 ppm
I20260924 07:11:35.549078 5736 webserver.cc:460] Webserver started at http://127.0.98.62:42737/ using document root <none> and password file <none>
I20260924 07:11:35.549714 5736 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:35.549789 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:35.549990 5736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:35.552130 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "a4bce354ac564f38ad1d0821157b7876"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "aeb2f4018906f6516d6e4da2ceda9ebe"
server_key_iv: "e51dc75cfc995d408cbe1bfeea74546d"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.552713 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4bce354ac564f38ad1d0821157b7876"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "aeb2f4018906f6516d6e4da2ceda9ebe"
server_key_iv: "e51dc75cfc995d408cbe1bfeea74546d"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.556506 5736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:35.559197 5750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:35.560703 5736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:35.560837 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "a4bce354ac564f38ad1d0821157b7876"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "aeb2f4018906f6516d6e4da2ceda9ebe"
server_key_iv: "e51dc75cfc995d408cbe1bfeea74546d"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.560959 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:35.579173 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:35.579932 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:35.580116 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:35.586700 5736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34455
I20260924 07:11:35.586722 5802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34455 every 8 connection(s)
I20260924 07:11:35.587635 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:35.590471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5736
I20260924 07:11:35.590584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:35.590870 392 external_mini_cluster.cc:1428] Setting key 8498de2ba32cdc7b47446788e4f0b494
I20260924 07:11:35.591126 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:35.597548 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876: Bootstrap starting.
I20260924 07:11:35.599820 5803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:35.599987 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.592265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.0.98.62:34455)
0924 07:11:35.592921 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.592937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.592973 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:35.593825 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.593845 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.594295 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.594610 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:35.595030 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.596038 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.596903 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.597367 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.597513 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.598305 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:35.598310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:35.598323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:35.598689 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:35.598744 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:35.598763 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:35.598845 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:35.598990 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:11:35.599240 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":137,"threads_started":1}
I20260924 07:11:35.600565 5803 log.cc:824] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:35.602491 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876: No bootstrap required, opened a new log
I20260924 07:11:35.602496 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:11:35.606047 5803 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 07:11:35.606271 5803 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:35.606338 5803 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4bce354ac564f38ad1d0821157b7876, State: Initialized, Role: FOLLOWER
I20260924 07:11:35.606781 5803 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 07:11:35.606912 5803 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:35.606971 5803 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:35.607091 5803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:35.608001 5803 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 07:11:35.608286 5803 leader_election.cc:302] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4bce354ac564f38ad1d0821157b7876; no voters:
I20260924 07:11:35.608532 5803 leader_election.cc:288] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:35.608635 5808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:35.608842 5808 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [term 1 LEADER]: Becoming Leader. State: Replica: a4bce354ac564f38ad1d0821157b7876, State: Running, Role: LEADER
I20260924 07:11:35.609131 5808 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } }
I20260924 07:11:35.609627 5803 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:35.610715 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4bce354ac564f38ad1d0821157b7876. Latest consensus state: current_term: 1 leader_uuid: "a4bce354ac564f38ad1d0821157b7876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260924 07:11:35.610715 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4bce354ac564f38ad1d0821157b7876" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4bce354ac564f38ad1d0821157b7876" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34455 } } }
I20260924 07:11:35.610852 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.610852 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.611280 5817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:35.614420 5817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:35.620080 5817 catalog_manager.cc:1347] Generated new cluster ID: b3e4a45ea34c410a90657ef687e299cd
I20260924 07:11:35.620142 5817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:35.625216 5817 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:35.626021 5817 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:35.634346 5817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4bce354ac564f38ad1d0821157b7876: Generated new TSK 0
I20260924 07:11:35.635108 5817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:11:35.651906 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.646968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51252 (local address 127.0.98.62:34455)
0924 07:11:35.647100 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.647105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.647254 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:11:35.647463 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.647465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.647519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.647616 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:11:35.647863 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.648413 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.649653 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.649871 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.649948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.650269 (+ 321us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:11:35.651547 (+ 1278us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:11:35.651788 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 07:11:35.651976 5806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51252: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 07:11:35.655056 5832 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:35.646829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34455 (local address 127.0.0.1:51252)
0924 07:11:35.647158 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:35.647173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:35.647297 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:35.647640 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:35.647643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:35.647652 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:11:35.647763 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.647769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.648521 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.648525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.649526 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.649536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.649959 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.649988 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.650077 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.650105 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:11:35.651709 (+ 1604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:11:35.654892 (+ 3183us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2778,"thread_start_us":121,"threads_started":1}
I20260924 07:11:35.655783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5736
E20260924 07:11:35.666306 5833 webserver.cc:533] Webserver: sslize failed:
E20260924 07:11:35.666407 5833 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T07:11:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T07:11:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:35Z Disabled control of system clock
I20260924 07:11:35.703598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33635
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33635
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/testchainca.cert with env {}
W20260924 07:11:35.815948 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:35.816251 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:35.816295 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:35.820461 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:35.820535 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:35.820549 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:35.820567 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:35.820583 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:11:35.820598 5837 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:11:35.820611 5837 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 07:11:35.826337 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:35.827536 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:35.828603 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:35.837687 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:35.837860 5837 server_base.cc:1034] running on GCE node
W20260924 07:11:35.837729 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:35.837776 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:35.838536 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:35.839787 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:35.840968 5837 hybrid_clock.cc:648] HybridClock initialized: now 1790233895840937 us; error 51 us; skew 500 ppm
I20260924 07:11:35.843055 5837 webserver.cc:460] Webserver started at http://127.0.98.62:39773/ using document root <none> and password file <none>
I20260924 07:11:35.843700 5837 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:35.843770 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:35.843957 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:35.846131 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "4547dd1597b44643a8efe24f908b69d1"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "0e4cc12a0e38c267eaecf0dce10d839f"
server_key_iv: "71a9834e0cc604ba5b5af70e035cec99"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.846726 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "4547dd1597b44643a8efe24f908b69d1"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "0e4cc12a0e38c267eaecf0dce10d839f"
server_key_iv: "71a9834e0cc604ba5b5af70e035cec99"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.850641 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:11:35.853370 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:35.854838 5837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:11:35.854972 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "4547dd1597b44643a8efe24f908b69d1"
format_stamp: "Formatted at 2026-09-24 07:11:35 on dist-test-slave-cnv3"
server_key: "0e4cc12a0e38c267eaecf0dce10d839f"
server_key_iv: "71a9834e0cc604ba5b5af70e035cec99"
server_key_version: "encryptionkey@0"
I20260924 07:11:35.855101 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:35.868876 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:35.869622 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:35.869817 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:35.876677 5837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33635
I20260924 07:11:35.876762 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33635 every 8 connection(s)
I20260924 07:11:35.877648 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:35.879137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5837
I20260924 07:11:35.879241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:35.879525 392 external_mini_cluster.cc:1428] Setting key 2466eb002412e84dc0c6daf6cb27a9b5
I20260924 07:11:35.881273 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:35.887097 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1: Bootstrap starting.
I20260924 07:11:35.888593 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.880860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.0.98.62:33635)
0924 07:11:35.881532 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.881548 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.881585 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:11:35.882243 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.882261 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.882681 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.883034 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:35.883509 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.884498 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.885367 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.885842 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.885997 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.886825 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:35.886830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:35.886843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:11:35.887240 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:35.887312 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:11:35.887339 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:35.887470 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:35.887608 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 07:11:35.887879 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260924 07:11:35.889555 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:35.890260 5904 log.cc:824] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:35.891289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:11:35.892433 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1: No bootstrap required, opened a new log
I20260924 07:11:35.895951 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } }
I20260924 07:11:35.896195 5904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:35.896245 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4547dd1597b44643a8efe24f908b69d1, State: Initialized, Role: FOLLOWER
I20260924 07:11:35.896615 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } }
I20260924 07:11:35.896721 5904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:35.896759 5904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:35.896826 5904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:35.897678 5904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } }
I20260924 07:11:35.897967 5904 leader_election.cc:302] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4547dd1597b44643a8efe24f908b69d1; no voters:
I20260924 07:11:35.898219 5904 leader_election.cc:288] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:35.898336 5909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:35.898564 5909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [term 1 LEADER]: Becoming Leader. State: Replica: 4547dd1597b44643a8efe24f908b69d1, State: Running, Role: LEADER
I20260924 07:11:35.898871 5909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } }
I20260924 07:11:35.899454 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:35.900548 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4547dd1597b44643a8efe24f908b69d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } } }
I20260924 07:11:35.900553 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4547dd1597b44643a8efe24f908b69d1. Latest consensus state: current_term: 1 leader_uuid: "4547dd1597b44643a8efe24f908b69d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4547dd1597b44643a8efe24f908b69d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33635 } } }
I20260924 07:11:35.900763 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.900792 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:35.901219 5919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:35.904054 5919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:35.909416 5919 catalog_manager.cc:1347] Generated new cluster ID: fedd6f343f1a4f7eb13b24b579a427f8
I20260924 07:11:35.909489 5919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:35.918874 5919 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:35.919973 5919 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:35.924155 5919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4547dd1597b44643a8efe24f908b69d1: Generated new TSK 0
I20260924 07:11:35.924825 5919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:11:36.048749 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:35.931499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.98.62:33635)
0924 07:11:35.931625 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:35.931629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:35.931756 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:11:35.931918 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:35.931921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:35.931979 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:35.932088 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:11:35.932296 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.933123 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.934029 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.934367 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.934466 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.934771 (+ 305us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:11:36.048355 (+113584us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:11:36.048621 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 07:11:36.048822 5907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34214: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 07:11:36.048930 5933 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:35.931365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33635 (local address 127.0.0.1:34214)
0924 07:11:35.931687 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:35.931700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:35.931794 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:35.932090 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:35.932092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:35.932098 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:11:35.932183 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.932188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.933285 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.933288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.933918 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:35.933926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:35.934483 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:35.934505 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:35.934575 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:35.934599 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:11:36.048576 (+113977us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:11:36.048750 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260924 07:11:36.049544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5837
2026-09-24T07:11:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (372 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:36 dist-test-slave-cnv3 krb5kdc[5945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:36 dist-test-slave-cnv3 krb5kdc[5945](info): set up 2 sockets
Sep 24 07:11:36 dist-test-slave-cnv3 krb5kdc[5945](info): commencing operation
krb5kdc: starting...
W20260924 07:11:37.255561 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:37 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:37.394662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:37.509246 5958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:37.509541 5958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:37.509600 5958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:37.513744 5958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:37.513823 5958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:37.513850 5958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:37.513871 5958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:37.513892 5958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:37.519498 5958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:37.520602 5958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:37.521636 5958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:37.530066 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:37.530160 5963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:37.530205 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:37.530313 5958 server_base.cc:1034] running on GCE node
I20260924 07:11:37.530886 5958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:37.532038 5958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:37.533198 5958 hybrid_clock.cc:648] HybridClock initialized: now 1790233897533174 us; error 40 us; skew 500 ppm
Sep 24 07:11:37 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:37.536036 5958 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:37.536967 5958 webserver.cc:460] Webserver started at http://127.0.98.62:45391/ using document root <none> and password file <none>
I20260924 07:11:37.537513 5958 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:37.537580 5958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:37.537767 5958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:37.539884 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "c7a00f824a044b11b25b243f683c2c6e"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "65775f80af3dfe02de1545b1097241f1"
server_key_iv: "302d4e35ee90bcd9b7503d43ddebd439"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.540447 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7a00f824a044b11b25b243f683c2c6e"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "65775f80af3dfe02de1545b1097241f1"
server_key_iv: "302d4e35ee90bcd9b7503d43ddebd439"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.544411 5958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:37.547004 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:37.548442 5958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:37.548579 5958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "c7a00f824a044b11b25b243f683c2c6e"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "65775f80af3dfe02de1545b1097241f1"
server_key_iv: "302d4e35ee90bcd9b7503d43ddebd439"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.548683 5958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:37.561606 5958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:37.564927 5958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:37.565125 5958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:37.572192 5958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36227
I20260924 07:11:37.572450 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36227 every 8 connection(s)
I20260924 07:11:37.573195 5958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:37.576041 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:37.581043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 5958
I20260924 07:11:37.581169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:37.581446 392 external_mini_cluster.cc:1428] Setting key 4f5d75aa8517d428f43f6f9b23586bdb
I20260924 07:11:37.581987 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e: Bootstrap starting.
I20260924 07:11:37.584484 6026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:37.585390 6026 log.cc:824] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:37.587678 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e: No bootstrap required, opened a new log
Sep 24 07:11:37 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:37.591886 6026 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } }
I20260924 07:11:37.592123 6026 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:37.592181 6026 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a00f824a044b11b25b243f683c2c6e, State: Initialized, Role: FOLLOWER
I20260924 07:11:37.592638 6026 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } }
I20260924 07:11:37.592780 6026 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:37.592854 6026 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:37.592983 6026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:37.594101 6026 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } }
I20260924 07:11:37.594437 6026 leader_election.cc:302] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a00f824a044b11b25b243f683c2c6e; no voters:
I20260924 07:11:37.594748 6026 leader_election.cc:288] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:37.594851 6031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:37.595052 6031 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [term 1 LEADER]: Becoming Leader. State: Replica: c7a00f824a044b11b25b243f683c2c6e, State: Running, Role: LEADER
I20260924 07:11:37.595413 6031 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } }
I20260924 07:11:37.596122 6026 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:37.596163 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:37.582917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.0.98.62:36227)
0924 07:11:37.583359 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:37.583399 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:37.583446 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:11:37.584113 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:37.584130 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:37.584483 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:37.584804 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:37.585693 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.586754 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.587920 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.588429 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.588553 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:37.591328 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:37.591334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:37.591348 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:37.591439 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:37.593194 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:37.593630 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:37.593634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:37.593639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:37.593748 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:37.594034 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:37.594037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:37.594038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:37.594464 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:37.594648 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:37.594954 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 07:11:37.595276 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260924 07:11:37.597086 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7a00f824a044b11b25b243f683c2c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } } }
I20260924 07:11:37.597278 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:37.597534 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7a00f824a044b11b25b243f683c2c6e. Latest consensus state: current_term: 1 leader_uuid: "c7a00f824a044b11b25b243f683c2c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a00f824a044b11b25b243f683c2c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36227 } } }
I20260924 07:11:37.597667 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:37.598327 6040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:37.601008 6040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:37.606606 6040 catalog_manager.cc:1347] Generated new cluster ID: d1c33ec5f3984a849d96efab644410d5
I20260924 07:11:37.606683 6040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:37.633773 6040 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:37.634778 6040 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:37.642915 6040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7a00f824a044b11b25b243f683c2c6e: Generated new TSK 0
I20260924 07:11:37.643684 6040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:37.679116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:37.793856 6052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:37.794162 6052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:37.794222 6052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:37.798524 6052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:37.798604 6052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:37.798666 6052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:37.804148 6052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:37.805215 6052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:37.806243 6052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:37.818010 6058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:37.818078 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:37.818073 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:37.818360 6052 server_base.cc:1034] running on GCE node
I20260924 07:11:37.818943 6052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:37.819651 6052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:37.820842 6052 hybrid_clock.cc:648] HybridClock initialized: now 1790233897820821 us; error 32 us; skew 500 ppm
Sep 24 07:11:37 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:37.823728 6052 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:37.824783 6052 webserver.cc:460] Webserver started at http://127.0.98.1:43583/ using document root <none> and password file <none>
I20260924 07:11:37.825358 6052 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:37.825430 6052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:37.825623 6052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:37.827803 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "91e170ec66be4b8881331a1e16a86e52"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "fcd2e4d074c449e7025241239d4ee8b9"
server_key_iv: "672d6068237024517f4e56fd3ab1691b"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.828388 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "91e170ec66be4b8881331a1e16a86e52"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "fcd2e4d074c449e7025241239d4ee8b9"
server_key_iv: "672d6068237024517f4e56fd3ab1691b"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.832633 6052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:37.835448 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:37.836843 6052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:37.837014 6052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "91e170ec66be4b8881331a1e16a86e52"
format_stamp: "Formatted at 2026-09-24 07:11:37 on dist-test-slave-cnv3"
server_key: "fcd2e4d074c449e7025241239d4ee8b9"
server_key_iv: "672d6068237024517f4e56fd3ab1691b"
server_key_version: "encryptionkey@0"
I20260924 07:11:37.837141 6052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:37.856117 6052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:37.859759 6052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:37.859977 6052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:37.860590 6052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:37.861443 6052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:37.861490 6052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:37.861531 6052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:37.861545 6052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:37.870596 6052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35797
I20260924 07:11:37.870625 6180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35797 every 8 connection(s)
I20260924 07:11:37.871575 6052 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:37.875643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6052
I20260924 07:11:37.875741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:37.875983 392 external_mini_cluster.cc:1428] Setting key d6f8cefa5eee63cd28786b09b764c293
Sep 24 07:11:37 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:37.886075 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:37.873455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47079 (local address 127.0.98.62:36227)
0924 07:11:37.873597 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:37.873601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:37.874432 (+ 831us) server_negotiation.cc:407] Connection header received
0924 07:11:37.875815 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:37.875818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:37.875873 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:37.875951 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:37.877364 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.878078 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.879542 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.879766 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.879865 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:37.882520 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:37.882523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:37.882525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:37.882548 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:37.883939 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:37.884430 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:37.884434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:37.884436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:37.884513 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:37.884868 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:37.884872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:37.884874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:37.885141 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:37.885238 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:37.885774 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 07:11:37.885909 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:11:37.886907 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:37.873852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36227 (local address 127.0.98.1:47079)
0924 07:11:37.874285 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:37.874328 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:37.875613 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:37.876139 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:37.876146 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:37.876584 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:37.877228 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:37.877249 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.878207 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.878210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:37.879422 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:37.879442 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:37.879915 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:37.879919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:37.880109 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:37.880707 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:37.880726 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:37.882342 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:37.884054 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:37.884060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:37.884076 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:37.884328 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:37.884600 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:37.884603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:37.884605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:37.884744 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:37.885262 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:37.885270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:37.885537 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:11:37.885715 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 07:11:37.886051 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:37.888273 6181 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36227
I20260924 07:11:37.888564 6181 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:37.889067 6181 heartbeater.cc:511] Master 127.0.98.62:36227 requested a full tablet report, sending...
I20260924 07:11:37.890837 5990 ts_manager.cc:194] Registered new tserver with Master: 91e170ec66be4b8881331a1e16a86e52 (127.0.98.1:35797)
I20260924 07:11:37.892393 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:37.906140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:38.015743 6186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:38.015995 6186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:38.016037 6186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:38.020112 6186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:38.020190 6186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:38.020249 6186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:38.026255 6186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:38.027407 6186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:38.028733 6186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:38.038632 6191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:38.038617 6192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:38.038619 6194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:38.038949 6186 server_base.cc:1034] running on GCE node
I20260924 07:11:38.039558 6186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:38.040261 6186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:38.041447 6186 hybrid_clock.cc:648] HybridClock initialized: now 1790233898041435 us; error 29 us; skew 500 ppm
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.044217 6186 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:38.045173 6186 webserver.cc:460] Webserver started at http://127.0.98.2:46367/ using document root <none> and password file <none>
I20260924 07:11:38.045758 6186 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:38.045828 6186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:38.046061 6186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:38.048252 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "2bcfe049d805481e98109ebc422891c5"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "1fa74d7d57ddcb62fb298275435b3cec"
server_key_iv: "5415e7a8ca8e1a629d0e505d64a31d89"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.048866 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2bcfe049d805481e98109ebc422891c5"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "1fa74d7d57ddcb62fb298275435b3cec"
server_key_iv: "5415e7a8ca8e1a629d0e505d64a31d89"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.052659 6186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:11:38.055154 6201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.056705 6186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:11:38.056813 6186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "2bcfe049d805481e98109ebc422891c5"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "1fa74d7d57ddcb62fb298275435b3cec"
server_key_iv: "5415e7a8ca8e1a629d0e505d64a31d89"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.056941 6186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:38.083853 6186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:38.088057 6186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:38.088259 6186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:38.088980 6186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:38.090032 6186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:38.090095 6186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.090154 6186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:38.090183 6186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.101065 6186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33281
I20260924 07:11:38.101081 6314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33281 every 8 connection(s)
I20260924 07:11:38.102180 6186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:38.102411 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6186
I20260924 07:11:38.102509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:38.102854 392 external_mini_cluster.cc:1428] Setting key 358d67577df7e148d103a85f697116c6
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:38.117591 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.105887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51643 (local address 127.0.98.62:36227)
0924 07:11:38.106033 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.106037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.106053 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:11:38.106974 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.106978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.107030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.107102 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.108468 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.108997 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.110731 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.110958 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.111058 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.113921 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:38.113925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:38.113927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:38.113952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:38.115300 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.115831 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.115834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.115836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.115929 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.116366 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.116371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.116373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.116623 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:38.116737 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:38.117250 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 07:11:38.117393 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:11:38.118541 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.104749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36227 (local address 127.0.98.2:51643)
0924 07:11:38.105305 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:38.105349 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:38.106759 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:38.107281 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:38.107290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:38.107789 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:38.108328 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:38.108342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.109124 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.109127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:38.110577 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:38.110589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.111139 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.111143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:38.111340 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.112018 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:38.112040 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:38.113747 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:38.115474 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:38.115479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:38.115493 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:38.115741 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:38.116044 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:38.116047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:38.116049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:38.116221 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:38.116733 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:38.116738 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:38.117005 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:11:38.117166 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 07:11:38.117447 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":148,"threads_started":1}
I20260924 07:11:38.119902 6315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36227
I20260924 07:11:38.120152 6315 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:38.120693 6315 heartbeater.cc:511] Master 127.0.98.62:36227 requested a full tablet report, sending...
I20260924 07:11:38.121797 5990 ts_manager.cc:194] Registered new tserver with Master: 2bcfe049d805481e98109ebc422891c5 (127.0.98.2:33281)
I20260924 07:11:38.122411 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:38.135016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:38.246948 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:38.247243 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:38.247287 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:38.251506 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:38.251580 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:38.251644 6320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:38.257186 6320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:38.258329 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:38.259486 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:38.270059 6325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:38.270051 6326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:38.270072 6328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:38.270222 6320 server_base.cc:1034] running on GCE node
I20260924 07:11:38.270792 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:38.271541 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:38.272744 6320 hybrid_clock.cc:648] HybridClock initialized: now 1790233898272711 us; error 47 us; skew 500 ppm
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.275683 6320 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:38.276635 6320 webserver.cc:460] Webserver started at http://127.0.98.3:42359/ using document root <none> and password file <none>
I20260924 07:11:38.277220 6320 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:38.277284 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:38.277484 6320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:38.279783 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "899ca93c3e8442e79b74b3f30ddd1e5e"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "9b7845b292dd83eb59292ee92b54ae3d"
server_key_iv: "d3c45954f6d91c052d77de01b1e25191"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.280417 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "899ca93c3e8442e79b74b3f30ddd1e5e"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "9b7845b292dd83eb59292ee92b54ae3d"
server_key_iv: "d3c45954f6d91c052d77de01b1e25191"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.284555 6320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:11:38.287168 6335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.288731 6320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:11:38.288849 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "899ca93c3e8442e79b74b3f30ddd1e5e"
format_stamp: "Formatted at 2026-09-24 07:11:38 on dist-test-slave-cnv3"
server_key: "9b7845b292dd83eb59292ee92b54ae3d"
server_key_iv: "d3c45954f6d91c052d77de01b1e25191"
server_key_version: "encryptionkey@0"
I20260924 07:11:38.288973 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:38.306874 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:38.310206 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:38.310380 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:38.311028 6320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:38.311956 6320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:38.312005 6320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.312047 6320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:38.312075 6320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:38.320832 6320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43819
I20260924 07:11:38.320860 6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43819 every 8 connection(s)
I20260924 07:11:38.321761 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:38.331697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6320
I20260924 07:11:38.331797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:38.332090 392 external_mini_cluster.cc:1428] Setting key b1526f98b8f7a9c1730304c3017e8417
I20260924 07:11:38.336021 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.323650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55259 (local address 127.0.98.62:36227)
0924 07:11:38.323820 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.323823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.324648 (+ 825us) server_negotiation.cc:407] Connection header received
0924 07:11:38.326029 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.326032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.326079 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.326162 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.327388 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.327985 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.329286 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.329507 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.329585 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.332253 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:38.332256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:38.332258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:38.332283 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:38.333826 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.334416 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.334420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.334438 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.334528 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.334898 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.334901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.334902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.335108 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:38.335189 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:38.335783 (+ 594us) server_negotiation.cc:299] Negotiation successful
0924 07:11:38.335890 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:11:38.336880 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.324024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36227 (local address 127.0.98.3:55259)
0924 07:11:38.324487 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:38.324531 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:38.325820 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:38.326325 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:38.326331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:38.326710 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:38.327205 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:38.327219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.328151 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.328154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:38.329162 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:38.329172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.329659 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.329663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:38.329816 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.330398 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:38.330414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:38.332069 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:38.333966 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:38.333971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:38.333985 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:38.334317 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:38.334643 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:38.334645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:38.334647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:38.334793 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:38.335223 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:38.335228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:38.335562 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 07:11:38.335732 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 07:11:38.336001 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260924 07:11:38.338084 6449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36227
I20260924 07:11:38.338320 6449 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:38.338822 6449 heartbeater.cc:511] Master 127.0.98.62:36227 requested a full tablet report, sending...
I20260924 07:11:38.339931 5990 ts_manager.cc:194] Registered new tserver with Master: 899ca93c3e8442e79b74b3f30ddd1e5e (127.0.98.3:43819)
I20260924 07:11:38.340529 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55259
I20260924 07:11:38.346412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/test-admin.keytab.
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.364413 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 07:11:38.367229 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:38.365793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.98.1:35797)
0924 07:11:38.366107 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.366121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.366146 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:11:38.366196 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.366201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.366422 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.366558 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.367076 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37822: BlockingRecv error: recv got EOF from 127.0.0.1:37822 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
W20260924 07:11:38.367444 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.368510 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.369475 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:38.379172 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.370395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37830 (local address 127.0.98.1:35797)
0924 07:11:38.370522 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.370568 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.370583 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:38.370682 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.370685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.370749 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.370814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.371410 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.372194 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.373018 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.373426 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.373521 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.375850 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:38.375859 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:38.375874 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:38.375899 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:38.377351 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.377782 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.377789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.377790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.377869 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.378124 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.378127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.378129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.378428 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:38.378637 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:38.378843 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:11:38.378994 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 07:11:38.382795 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:38.381964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.98.1:35797)
0924 07:11:38.382096 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.382099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.382161 (+ 62us) server_negotiation.cc:407] Connection header received
0924 07:11:38.382247 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.382249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.382283 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.382346 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.382710 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37842: BlockingRecv error: recv got EOF from 127.0.0.1:37842 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 07:11:38.383167 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc)
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.384177 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.385061 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:38.393231 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.385856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.0.98.1:35797)
0924 07:11:38.385985 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.385988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.386051 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:11:38.386140 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.386142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.386176 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.386236 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.386648 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.387161 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.388066 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.388298 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.388375 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.390461 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:38.390464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:38.390466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:38.390489 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:38.391799 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.392210 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.392214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.392215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.392292 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.392480 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.392482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.392483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.392666 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:38.392789 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:38.392974 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 07:11:38.393069 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 07:11:38.395843 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:38.394952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.0.98.1:35797)
0924 07:11:38.395086 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.395089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.395162 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:11:38.395252 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.395254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.395287 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.395342 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.395754 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37870: BlockingRecv error: recv got EOF from 127.0.0.1:37870 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 07:11:38.395956 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233866113376-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.396863 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:38.397748 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:11:38.406106 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:38.398630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.0.98.1:35797)
0924 07:11:38.398747 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:38.398750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:38.398763 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:38.398859 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:38.398861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:38.398894 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:38.398951 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:38.399362 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.399922 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.400667 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:38.400873 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:38.400945 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:38.403034 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:38.403037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:38.403039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:38.403061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:38.404535 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.405001 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.405012 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.405014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.405093 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:38.405335 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:38.405338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:38.405340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:38.405513 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:38.405624 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:38.405845 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 07:11:38.405942 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:11:38.407094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6052
I20260924 07:11:38.412264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6186
I20260924 07:11:38.416738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6320
I20260924 07:11:38.421211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 5958
2026-09-24T07:11:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2128 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[6469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[6469](info): set up 2 sockets
Sep 24 07:11:38 dist-test-slave-cnv3 krb5kdc[6469](info): commencing operation
krb5kdc: starting...
W20260924 07:11:39.375573 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:39 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 07:11:39.499436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:39.611212 6482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:39.611513 6482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:39.611572 6482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:39.615722 6482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:39.615788 6482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:39.615814 6482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:39.615845 6482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:39.615872 6482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:39.621321 6482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:39.622377 6482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:39.623427 6482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:39.631711 6488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:39.631711 6487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:39.631793 6482 server_base.cc:1034] running on GCE node
W20260924 07:11:39.631714 6490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:39.632592 6482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:39.633767 6482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:39.634960 6482 hybrid_clock.cc:648] HybridClock initialized: now 1790233899634894 us; error 78 us; skew 500 ppm
Sep 24 07:11:39 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:39.641263 6482 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 07:11:39.642287 6482 webserver.cc:460] Webserver started at http://127.0.98.62:41849/ using document root <none> and password file <none>
I20260924 07:11:39.642853 6482 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:39.642910 6482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:39.643095 6482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:39.645259 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "f88fc2058d034f66a45c9b616dc7e4a4"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "f227d9b58b9018f1479978cd275bfeff"
server_key_iv: "8fe307db645e8217220e52b8f481b298"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.645777 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "f88fc2058d034f66a45c9b616dc7e4a4"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "f227d9b58b9018f1479978cd275bfeff"
server_key_iv: "8fe307db645e8217220e52b8f481b298"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.649554 6482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:39.651997 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:39.653373 6482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:39.653488 6482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "f88fc2058d034f66a45c9b616dc7e4a4"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "f227d9b58b9018f1479978cd275bfeff"
server_key_iv: "8fe307db645e8217220e52b8f481b298"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.653590 6482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:39.665273 6482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:39.668677 6482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:39.668869 6482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:39.675710 6482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45079
I20260924 07:11:39.675971 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45079 every 8 connection(s)
I20260924 07:11:39.676687 6482 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:39.679755 6550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:39.685551 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Bootstrap starting.
I20260924 07:11:39.685912 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6482
I20260924 07:11:39.686022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:39.686326 392 external_mini_cluster.cc:1428] Setting key d80df39fa1ba32db6db352e70d71d4d5
I20260924 07:11:39.688365 6550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:39.689278 6550 log.cc:824] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:39.691285 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: No bootstrap required, opened a new log
I20260924 07:11:39.694818 6550 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260924 07:11:39.695065 6550 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:39.695113 6550 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88fc2058d034f66a45c9b616dc7e4a4, State: Initialized, Role: FOLLOWER
Sep 24 07:11:39 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:39.695612 6550 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260924 07:11:39.695766 6550 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:39.695835 6550 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:39.695938 6550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:39.697023 6550 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260924 07:11:39.697379 6550 leader_election.cc:302] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88fc2058d034f66a45c9b616dc7e4a4; no voters:
I20260924 07:11:39.697736 6550 leader_election.cc:288] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:39.697846 6555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:39.698055 6555 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 LEADER]: Becoming Leader. State: Replica: f88fc2058d034f66a45c9b616dc7e4a4, State: Running, Role: LEADER
I20260924 07:11:39.698387 6555 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } }
I20260924 07:11:39.699075 6550 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:39.700093 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } }
I20260924 07:11:39.700305 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:39.700613 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88fc2058d034f66a45c9b616dc7e4a4. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } }
I20260924 07:11:39.700760 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:39.701231 6564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:39.702322 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:39.687776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.98.62:45079)
0924 07:11:39.688265 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:39.688286 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:39.688331 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:11:39.689209 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:39.689234 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:39.689698 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:39.690038 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:39.690886 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.691950 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.692809 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.693298 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.693448 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:39.695972 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:39.695977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:39.695992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:39.696030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:39.698162 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:39.698717 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:39.698723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:39.698729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:39.698873 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:39.699222 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:39.699225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:39.699227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:39.699810 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:39.700639 (+ 829us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:39.700816 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:11:39.701302 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":132,"threads_started":1}
I20260924 07:11:39.704386 6564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:39.710338 6564 catalog_manager.cc:1347] Generated new cluster ID: e446a3acfb6f46f8b7ee46d434beaab3
I20260924 07:11:39.710420 6564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:39.737269 6564 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:39.738511 6564 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:39.753371 6564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Generated new TSK 0
I20260924 07:11:39.754132 6564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 07:11:39.781973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:39.893277 6576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:39.893551 6576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:39.893594 6576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:39.897701 6576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:39.897768 6576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:39.897830 6576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:39.903331 6576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:39.904441 6576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:39.905529 6576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:39.915108 6584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:39.915122 6582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:39.915105 6581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:39.915447 6576 server_base.cc:1034] running on GCE node
I20260924 07:11:39.915877 6576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:39.916460 6576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:39.917622 6576 hybrid_clock.cc:648] HybridClock initialized: now 1790233899917598 us; error 36 us; skew 500 ppm
Sep 24 07:11:39 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:39.920410 6576 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:11:39.921268 6576 webserver.cc:460] Webserver started at http://127.0.98.1:42187/ using document root <none> and password file <none>
I20260924 07:11:39.921794 6576 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:39.921856 6576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:39.922039 6576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:39.924072 6576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b3a300362fc43e0ae7cda13d60ba0c3"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "1f1f3ee2e2942f4fb3296a4d404dd9a5"
server_key_iv: "a3e9c38d7c0016de00dbe917d2cc84e9"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.924587 6576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b3a300362fc43e0ae7cda13d60ba0c3"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "1f1f3ee2e2942f4fb3296a4d404dd9a5"
server_key_iv: "a3e9c38d7c0016de00dbe917d2cc84e9"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.928229 6576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:39.930589 6591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:39.932155 6576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:39.932283 6576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "5b3a300362fc43e0ae7cda13d60ba0c3"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "1f1f3ee2e2942f4fb3296a4d404dd9a5"
server_key_iv: "a3e9c38d7c0016de00dbe917d2cc84e9"
server_key_version: "encryptionkey@0"
I20260924 07:11:39.932390 6576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:39.947813 6576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:39.951086 6576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:39.951273 6576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:39.951900 6576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:39.952731 6576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:39.952786 6576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:39.952841 6576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:39.952867 6576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:39.961314 6576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37029
I20260924 07:11:39.961349 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37029 every 8 connection(s)
I20260924 07:11:39.962249 6576 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:39.968569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6576
I20260924 07:11:39.968701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:39.968983 392 external_mini_cluster.cc:1428] Setting key 353514c8c8be0565990340676a67f38f
Sep 24 07:11:39 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:39.977807 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:39.964292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55487 (local address 127.0.98.62:45079)
0924 07:11:39.964419 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:39.964423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:39.965316 (+ 893us) server_negotiation.cc:407] Connection header received
0924 07:11:39.966696 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:39.966699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:39.966785 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:39.966856 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:39.968164 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.968881 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.970315 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.970651 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.970728 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:39.973943 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:39.973947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:39.973950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:39.973980 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:39.975524 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:39.976077 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:39.976081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:39.976083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:39.976182 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:39.976526 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:39.976529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:39.976531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:39.976788 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:39.976888 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:39.977479 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 07:11:39.977615 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:39.978662 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:39.964636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:55487)
0924 07:11:39.965113 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:39.965164 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:39.966467 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:39.967059 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:39.967066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:39.967513 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:39.968013 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:39.968028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.969032 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.969036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:39.970179 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:39.970197 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:39.970949 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:39.970953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:39.971159 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:39.971830 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:39.971852 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:39.973737 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:39.975667 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:39.975673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:39.975688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:39.975969 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:39.976281 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:39.976283 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:39.976285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:39.976434 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:39.976919 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:39.976926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:39.977218 (+ 292us) client_negotiation.cc:769] Sending connection context
0924 07:11:39.977399 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 07:11:39.977730 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:39.979979 6705 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:39.980247 6705 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:39.980769 6705 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:39.982523 6514 ts_manager.cc:194] Registered new tserver with Master: 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1:37029)
I20260924 07:11:39.983893 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 07:11:39.998307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:40.113169 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:40.113474 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:40.113535 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:40.117702 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:40.117776 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:40.117858 6710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:40.123410 6710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:40.124529 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:40.125599 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:40.135412 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.135528 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.135604 6710 server_base.cc:1034] running on GCE node
W20260924 07:11:40.135414 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.136184 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:40.136981 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:40.138166 6710 hybrid_clock.cc:648] HybridClock initialized: now 1790233900138129 us; error 50 us; skew 500 ppm
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:40.140946 6710 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:11:40.141815 6710 webserver.cc:460] Webserver started at http://127.0.98.2:34233/ using document root <none> and password file <none>
I20260924 07:11:40.142378 6710 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.142441 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.142606 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:40.144708 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "30ce379435aa4f5e8b550c50cd93009c"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "728bc72213f2ba5d5de0d00f4b49d45c"
server_key_iv: "7020effa0b66ab8ae8032f7e4f4d4789"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.145229 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30ce379435aa4f5e8b550c50cd93009c"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "728bc72213f2ba5d5de0d00f4b49d45c"
server_key_iv: "7020effa0b66ab8ae8032f7e4f4d4789"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.149011 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:40.151597 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.152948 6710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:40.153054 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "30ce379435aa4f5e8b550c50cd93009c"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "728bc72213f2ba5d5de0d00f4b49d45c"
server_key_iv: "7020effa0b66ab8ae8032f7e4f4d4789"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.153168 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.185818 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:40.188964 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:40.189159 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:40.189761 6710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:40.190621 6710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:40.190667 6710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.190706 6710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:40.190722 6710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.198942 6710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35867
I20260924 07:11:40.198974 6838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35867 every 8 connection(s)
I20260924 07:11:40.199934 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:40.205399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6710
I20260924 07:11:40.205511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:40.205832 392 external_mini_cluster.cc:1428] Setting key 58a1ed0839d8907777cafa256163fe76
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:40.214887 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.201924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54365 (local address 127.0.98.62:45079)
0924 07:11:40.202042 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.202045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.202928 (+ 883us) server_negotiation.cc:407] Connection header received
0924 07:11:40.204302 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.204305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.204355 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.204427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.205827 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.206527 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.208097 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.208318 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.208398 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.211202 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.211207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.211210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.211242 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.212821 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.213299 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.213303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.213305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.213401 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.213710 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.213713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.213715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.213957 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.214063 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.214566 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.214701 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:40.215764 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.202324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:54365)
0924 07:11:40.202767 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.202813 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.204065 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.204592 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.204599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.204959 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.205647 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.205671 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.206645 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.206649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.207980 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.207992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.208463 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.208466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.208658 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.209391 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.209407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.211008 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.212935 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.212941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.212955 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.213203 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.213488 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.213490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.213491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.213622 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.214051 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.214056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.214310 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.214479 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.214744 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260924 07:11:40.217238 6839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:40.217514 6839 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:40.217993 6839 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:40.219084 6514 ts_manager.cc:194] Registered new tserver with Master: 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:40.219800 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:11:40.234846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:40.343777 6844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:40.344048 6844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:40.344089 6844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:40.348131 6844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:40.348203 6844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:40.348263 6844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:40.353533 6844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:40.354605 6844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:40.355664 6844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:40.365710 6852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.365725 6850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.365711 6849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.365959 6844 server_base.cc:1034] running on GCE node
I20260924 07:11:40.366556 6844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:40.367246 6844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:40.368443 6844 hybrid_clock.cc:648] HybridClock initialized: now 1790233900368422 us; error 33 us; skew 500 ppm
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:40.371368 6844 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:11:40.372377 6844 webserver.cc:460] Webserver started at http://127.0.98.3:33605/ using document root <none> and password file <none>
I20260924 07:11:40.372922 6844 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.372982 6844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.373162 6844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:40.375217 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "8147ec5745404904be9e1c1f9ddc2de7"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "f63b4a1ad0fead4fcf9f88f4c830bb66"
server_key_iv: "11e59667d001a1ff2b39565be9ab3603"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.375830 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8147ec5745404904be9e1c1f9ddc2de7"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "f63b4a1ad0fead4fcf9f88f4c830bb66"
server_key_iv: "11e59667d001a1ff2b39565be9ab3603"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.379704 6844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:40.382151 6859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.383447 6844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:40.383561 6844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "8147ec5745404904be9e1c1f9ddc2de7"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "f63b4a1ad0fead4fcf9f88f4c830bb66"
server_key_iv: "11e59667d001a1ff2b39565be9ab3603"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.383670 6844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.399408 6844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:40.402555 6844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:40.402729 6844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:40.403338 6844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:40.404232 6844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:40.404285 6844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.404326 6844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:40.404354 6844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.412679 6844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45263
I20260924 07:11:40.412709 6972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45263 every 8 connection(s)
I20260924 07:11:40.413569 6844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:40.421247 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6844
I20260924 07:11:40.421377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:40.421669 392 external_mini_cluster.cc:1428] Setting key dc116030fad48765e5b5a2dee21a914c
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:40.427500 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.415344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37341 (local address 127.0.98.62:45079)
0924 07:11:40.415481 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.415484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.416474 (+ 990us) server_negotiation.cc:407] Connection header received
0924 07:11:40.417783 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.417786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.417833 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.417911 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.419114 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.419799 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.421098 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.421416 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.421521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.424178 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.424182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.424184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.424213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.425477 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.425969 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.425973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.425975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.426054 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.426373 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.426376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.426378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.426568 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.426664 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.427171 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.427307 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:11:40.428349 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.415741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:37341)
0924 07:11:40.416236 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.416364 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.417598 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.418080 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.418088 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.418455 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.418945 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.418958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.419952 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.419955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.420965 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.420975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.421582 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.421586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.421755 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.422356 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.422372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.424023 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.425589 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.425595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.425612 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.425871 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.426138 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.426141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.426143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.426285 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.426668 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.426676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.426942 (+ 266us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.427112 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.427495 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":116,"threads_started":1}
I20260924 07:11:40.429491 6973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:40.429735 6973 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:40.430323 6973 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
I20260924 07:11:40.431445 6514 ts_manager.cc:194] Registered new tserver with Master: 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
I20260924 07:11:40.432055 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37341
I20260924 07:11:40.436105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 07:11:40.466962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38141
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45079,127.0.98.61:38141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:40.579869 6978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:40.580147 6978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:40.580188 6978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:40.584651 6978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:40.584733 6978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:40.584753 6978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:40.584769 6978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:40.584784 6978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:40.590353 6978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45079,127.0.98.61:38141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
W20260924 07:11:40.590552 6978 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45079,127.0.98.61:38141'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 07:11:40.591429 6978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:40.592451 6978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:40.600739 6983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.600739 6986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.600744 6984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.601006 6978 server_base.cc:1034] running on GCE node
I20260924 07:11:40.601712 6978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:40.603001 6978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:40.604156 6978 hybrid_clock.cc:648] HybridClock initialized: now 1790233900604138 us; error 33 us; skew 500 ppm
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:40.606801 6978 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 07:11:40.607725 6978 webserver.cc:460] Webserver started at http://127.0.98.61:45499/ using document root <none> and password file <none>
I20260924 07:11:40.608248 6978 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.608311 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.608529 6978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:40.610517 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/instance:
uuid: "08d3f7f2168045999a923578819a9207"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "a5eade545d8b99c36c0651bc2923363a"
server_key_iv: "f703ef3845b6857b3dfcf78dea809c9d"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.611061 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal/instance:
uuid: "08d3f7f2168045999a923578819a9207"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "a5eade545d8b99c36c0651bc2923363a"
server_key_iv: "f703ef3845b6857b3dfcf78dea809c9d"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.614778 6978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:11:40.617264 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.618623 6978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:11:40.618737 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "08d3f7f2168045999a923578819a9207"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "a5eade545d8b99c36c0651bc2923363a"
server_key_iv: "f703ef3845b6857b3dfcf78dea809c9d"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.618835 6978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.639668 6978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:40.643042 6978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:40.643473 6978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:40.650588 6978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38141
I20260924 07:11:40.650851 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38141 every 8 connection(s)
I20260924 07:11:40.651633 6978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:11:40.653766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 6978
I20260924 07:11:40.654541 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:40.658977 7046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:40.668995 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.655132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.98.61:38141)
0924 07:11:40.655551 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.655566 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.655603 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:11:40.656271 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.656288 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.656703 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.657063 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.657911 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.658977 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.660121 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.660671 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.660830 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.663838 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.663843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.663853 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.663902 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.665716 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.666206 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.666213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.666220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.666362 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.666595 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.666598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.666600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.667095 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.667280 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.667594 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.667899 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260924 07:11:40.670982 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.660963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47482 (local address 127.0.98.62:45079)
0924 07:11:40.661107 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.661111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.661466 (+ 355us) server_negotiation.cc:407] Connection header received
0924 07:11:40.661599 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.661601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.661652 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.661731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.662583 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.663351 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.664470 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.664789 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.664882 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.667433 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.667437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.667440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.667470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.668996 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.669545 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.669549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.669551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.669648 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.669965 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.669968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.669970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.670262 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.670357 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.670754 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.670860 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:11:40.671077 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.660898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47482)
0924 07:11:40.661258 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.661379 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.661503 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.661804 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.661811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.662273 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.662453 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.662462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.663508 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.663512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.664300 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.664312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.664916 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.664920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.665020 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.665428 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.665440 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.667255 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.669142 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.669158 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.669165 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.669438 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.669763 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.669766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.669767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.669876 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.670414 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.670420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.670555 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.670695 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.670888 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":100,"threads_started":1}
I20260924 07:11:40.672628 7046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:11:40.680372 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.673108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43892)
0924 07:11:40.673218 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.673231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.673295 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.673478 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.673480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.673611 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.673700 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.673705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.674738 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.674740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.675263 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.675270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.675731 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.675735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.675831 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.676265 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.676278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.677547 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.678987 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.678989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.678992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.679161 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.679461 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.679463 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.679465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.679540 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.679885 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.679888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.679971 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.680054 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.680192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:11:40.680372 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.673167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.61:38141)
0924 07:11:40.673292 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.673295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.673307 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:40.673363 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.673365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.673396 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.673461 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.673777 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.674651 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.675417 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.675633 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.675709 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.677661 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.677664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.677666 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.677684 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.678886 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.679236 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.679239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.679241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.679303 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.679613 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.679616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.679618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.679782 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.679875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.680102 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.680245 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":53}
I20260924 07:11:40.683931 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Bootstrap starting.
I20260924 07:11:40.686655 7046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:40.687608 7046 log.cc:824] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:40.689764 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: No bootstrap required, opened a new log
I20260924 07:11:40.692811 7046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } }
I20260924 07:11:40.693023 7046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:40.693058 7046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d3f7f2168045999a923578819a9207, State: Initialized, Role: FOLLOWER
I20260924 07:11:40.693466 7046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } }
I20260924 07:11:40.693971 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } } }
I20260924 07:11:40.694119 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:40.694619 7046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:11:40.699425 7063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:11:40.699566 7063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:11:40.701983 6978 master_runner.cc:428] Detected that this master 08d3f7f2168045999a923578819a9207 is joining an existing cluster
I20260924 07:11:40.702060 6978 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38141
I20260924 07:11:40.712090 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.704594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.0.98.62:45079)
0924 07:11:40.704728 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.704733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.704868 (+ 135us) server_negotiation.cc:407] Connection header received
0924 07:11:40.704981 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.704984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.705035 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.705126 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.705684 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.706219 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.707354 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.707607 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.707693 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.708833 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.708837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.708840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.708864 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.710443 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.710879 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.710883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.710885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.710975 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.711254 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.711257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.711259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.711512 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.711599 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.711830 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.711934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:11:40.712118 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.704499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47484)
0924 07:11:40.704770 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.704783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.704895 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.705140 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.705143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.705426 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.705539 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.705545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.706322 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.706325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.707269 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.707277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.707720 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.707723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.707807 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.708272 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.708286 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.708698 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.710545 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.710548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.710550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.710792 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.711064 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.711067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.711069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.711152 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.711598 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.711600 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.711690 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.711771 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.711913 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260924 07:11:40.712934 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.705222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43894)
0924 07:11:40.705497 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.705510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.705614 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.705913 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.705915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.706136 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.706278 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.706295 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.707299 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.707303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.708234 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.708243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.708776 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.708780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.708872 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.709363 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.709377 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.709779 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.711467 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.711470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.711472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.711745 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.712000 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.712002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.712004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.712091 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.712469 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.712472 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.712538 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.712626 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.712745 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"spinlock_wait_cycles":2816,"thread_start_us":84,"threads_started":1}
I20260924 07:11:40.712953 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.705084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.0.98.61:38141)
0924 07:11:40.705175 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.705179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.705588 (+ 409us) server_negotiation.cc:407] Connection header received
0924 07:11:40.705711 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.705714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.705750 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.705792 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.706385 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.707191 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.708349 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.708653 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.708750 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.709908 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.709911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.709914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.709938 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.711365 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.711845 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.711848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.711850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.711917 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.712180 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.712182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.712184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.712357 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.712419 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.712695 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.712799 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":38}
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:11:40.726799 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.717199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43896 (local address 127.0.98.61:38141)
0924 07:11:40.717321 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.717325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.717605 (+ 280us) server_negotiation.cc:407] Connection header received
0924 07:11:40.717781 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.717783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.717826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.717893 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.718772 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.719278 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.720489 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.720693 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.720767 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.723321 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.723325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.723328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.723353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.724897 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.725453 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.725456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.725458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.725540 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.725796 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.725799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.725801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.726054 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.726145 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.726511 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.726634 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:40.726841 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.717137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43896)
0924 07:11:40.717476 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.717530 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.717664 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.717939 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.717945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.718514 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.718663 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.718671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.719364 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.719367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.720389 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.720399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.720785 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.720789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.720903 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.721393 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.721407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.723187 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.725020 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.725030 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.725037 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.725343 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.725622 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.725625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.725627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.725717 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.726152 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.726159 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.726302 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.726449 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.726654 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260924 07:11:40.728832 6514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38141: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08d3f7f2168045999a923578819a9207"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38141
}
attrs {
promote: true
}
}
dest_uuid: "f88fc2058d034f66a45c9b616dc7e4a4"
cas_config_opid_index: -1
I20260924 07:11:40.729671 6514 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 07:11:40.731634 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08d3f7f2168045999a923578819a9207 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } }
I20260924 07:11:40.732662 7020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:40.733912 7020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 07:11:40.734035 7074 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38141
I20260924 07:11:40.734264 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260924 07:11:40.734369 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:40.734607 6978 master.cc:537] Master@127.0.98.61:38141 shutting down...
I20260924 07:11:40.736244 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:11:40.738591 6499 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 07:11:40.739257 6978 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:40.739672 6978 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:11:40.739810 6499 consensus_peers.cc:487] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 -> Peer 08d3f7f2168045999a923578819a9207 (127.0.98.61:38141): Couldn't send request to peer 08d3f7f2168045999a923578819a9207. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:11:40.739938 6978 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: stopping tablet replica
I20260924 07:11:40.756174 6978 master.cc:552] Master@127.0.98.61:38141 shutdown complete.
I20260924 07:11:40.756296 6978 master_runner.cc:305] Clearing existing system tablet
I20260924 07:11:40.756524 6978 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.756573 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.757925 6978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:11:40.758950 7079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.759177 6978 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.759317 6978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:40.759359 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "08d3f7f2168045999a923578819a9207"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "a5eade545d8b99c36c0651bc2923363a"
server_key_iv: "f703ef3845b6857b3dfcf78dea809c9d"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 07:11:40.761839 6978 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:40.762465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33473
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:11:40.764365 6978 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:11:40.764460 6978 log.cc:1196] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 07:11:40.764714 6978 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Deleting consensus metadata
I20260924 07:11:40.764885 6978 master_runner.cc:315] Copying system tablet from 127.0.98.62:45079
I20260924 07:11:40.766105 6978 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45079
I20260924 07:11:40.775273 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.766869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.98.62:45079)
0924 07:11:40.767005 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.767009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.767145 (+ 136us) server_negotiation.cc:407] Connection header received
0924 07:11:40.767325 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.767328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.767421 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.767512 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.768052 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.768868 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.770105 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.770481 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.770594 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.771945 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.771949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.771952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.771982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.773590 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.774071 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.774074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.774077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.774159 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.774430 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.774433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.774434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.774646 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.774735 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.774994 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.775110 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:11:40.775307 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.766739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47494)
0924 07:11:40.767051 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.767065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.767184 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.767554 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.767557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.767795 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.767925 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.767933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.769010 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.769013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.769966 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.769978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.770613 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.770616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.770719 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.771322 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.771336 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.771788 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.773700 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.773704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.773706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.773969 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.774251 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.774254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.774256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.774349 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.774731 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.774733 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.774834 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.774936 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.775086 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":88,"threads_started":1}
I20260924 07:11:40.776155 6534 tablet_copy_service.cc:140] P f88fc2058d034f66a45c9b616dc7e4a4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08d3f7f2168045999a923578819a9207 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47494)
I20260924 07:11:40.776320 6534 tablet_copy_service.cc:161] P f88fc2058d034f66a45c9b616dc7e4a4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08d3f7f2168045999a923578819a9207 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47494: session id = 08d3f7f2168045999a923578819a9207-00000000000000000000000000000000
I20260924 07:11:40.777693 6534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:11:40.779456 6978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:11:40.782028 6978 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: tablet copy: Starting download of 0 data blocks...
I20260924 07:11:40.782238 6978 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: tablet copy: Starting download of 1 WAL segments...
I20260924 07:11:40.784215 6978 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:11:40.786151 6534 tablet_copy_service.cc:342] P f88fc2058d034f66a45c9b616dc7e4a4: Request end of tablet copy session 08d3f7f2168045999a923578819a9207-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47494
I20260924 07:11:40.786305 6534 tablet_copy_service.cc:433] P f88fc2058d034f66a45c9b616dc7e4a4: ending tablet copy session 08d3f7f2168045999a923578819a9207-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08d3f7f2168045999a923578819a9207
I20260924 07:11:40.794448 6978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:40.796350 7087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.796597 7090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.796669 6978 server_base.cc:1034] running on GCE node
W20260924 07:11:40.797055 7088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.797284 6978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:40.797595 6978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:40.798717 6978 hybrid_clock.cc:648] HybridClock initialized: now 1790233900798695 us; error 38 us; skew 500 ppm
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:40.800397 6978 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 07:11:40.800839 6978 webserver.cc:460] Webserver started at http://127.0.98.61:36525/ using document root <none> and password file <none>
I20260924 07:11:40.801016 6978 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.801054 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.806813 6978 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 07:11:40.807945 7097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.808239 6978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:40.808322 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "08d3f7f2168045999a923578819a9207"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "a5eade545d8b99c36c0651bc2923363a"
server_key_iv: "f703ef3845b6857b3dfcf78dea809c9d"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.808411 6978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.821424 6978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:40.823664 6978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:40.828536 6978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38141
I20260924 07:11:40.828560 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38141 every 8 connection(s)
I20260924 07:11:40.829146 6978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:11:40.831763 7150 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:11:40.832868 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Bootstrap starting.
I20260924 07:11:40.843850 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:11:40.844198 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Bootstrap complete.
I20260924 07:11:40.844792 7150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } }
I20260924 07:11:40.844930 7150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08d3f7f2168045999a923578819a9207, State: Initialized, Role: LEARNER
I20260924 07:11:40.845027 7150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } }
I20260924 07:11:40.845369 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } }
I20260924 07:11:40.845502 7150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:40.845577 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:11:40.848898 7164 catalog_manager.cc:1259] Loaded cluster ID: e446a3acfb6f46f8b7ee46d434beaab3
I20260924 07:11:40.848948 7164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: loading cluster ID for follower catalog manager: success
I20260924 07:11:40.850529 7164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: acquiring CA information for follower catalog manager: success
I20260924 07:11:40.851730 7164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:11:40.899221 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:40.899513 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:40.899562 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:40.903789 7080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:40.903853 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:40.903872 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:40.903889 7080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:40.903903 7080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:40.909289 7080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:40.910409 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:40.911412 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:11:40.919824 7080 server_base.cc:1034] running on GCE node
W20260924 07:11:40.919770 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.919770 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:40.919767 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:40.920610 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:40.921706 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:40.922883 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790233900922870 us; error 31 us; skew 500 ppm
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:40.925657 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 07:11:40.926512 7080 webserver.cc:460] Webserver started at http://127.0.98.60:35545/ using document root <none> and password file <none>
I20260924 07:11:40.927017 7080 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:40.927071 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:40.927238 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:40.929291 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/instance:
uuid: "8b670153be384e569c195a408e963efe"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "738347d543938ee45a3e6666238b93f4"
server_key_iv: "faaa64b9051f73c25f7a792c2b4ffbb1"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.929783 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal/instance:
uuid: "8b670153be384e569c195a408e963efe"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "738347d543938ee45a3e6666238b93f4"
server_key_iv: "faaa64b9051f73c25f7a792c2b4ffbb1"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.933514 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260924 07:11:40.935899 7179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:40.937177 7080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:40.937278 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "8b670153be384e569c195a408e963efe"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "738347d543938ee45a3e6666238b93f4"
server_key_iv: "faaa64b9051f73c25f7a792c2b4ffbb1"
server_key_version: "encryptionkey@0"
I20260924 07:11:40.937371 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:40.962486 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:40.965757 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:40.965934 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:40.973722 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33473
I20260924 07:11:40.973989 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33473 every 8 connection(s)
I20260924 07:11:40.974720 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:11:40.977541 7232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:40.979405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7080
I20260924 07:11:40.981722 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:11:40.986555 6705 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:40.992522 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.983886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.98.62:45079)
0924 07:11:40.984014 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.984018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.984150 (+ 132us) server_negotiation.cc:407] Connection header received
0924 07:11:40.984265 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.984268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.984316 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.984393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.985032 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.985545 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.986490 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.986800 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.986873 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.989608 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.989612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.989614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.989635 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.990820 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.991233 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.991236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.991238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.991308 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.991604 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.991608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.991610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.991786 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.991870 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.992241 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.992356 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:11:40.993351 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.983705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47506)
0924 07:11:40.984031 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.984078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.984186 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.984450 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.984454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.984784 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.984911 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.984918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.985659 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.985661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.986329 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.986358 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.987025 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.987028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.987112 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.987665 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.987685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:40.989484 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:40.990922 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.990930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.990940 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.991146 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.991409 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:40.991412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:40.991414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:40.991523 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:40.991876 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:40.991891 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:40.992045 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 07:11:40.992188 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 07:11:40.992498 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":6,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260924 07:11:40.993356 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.980914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.98.60:33473)
0924 07:11:40.981379 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.981398 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.981443 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:11:40.982101 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.982118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.982486 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.982799 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.983614 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.984844 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.985658 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.986189 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.986339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.989148 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:40.989155 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:40.989168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:40.989193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:40.990863 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.991267 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.991273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.991275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.991406 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:40.991646 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:40.991649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:40.991652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:40.992027 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:40.992196 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:40.992416 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 07:11:40.992551 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":309,"thread_start_us":135,"threads_started":1}
I20260924 07:11:40.995141 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:11:40 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:11:41.003301 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.995604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43902)
0924 07:11:40.995739 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:40.995753 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:40.995828 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:40.996264 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:40.996266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:40.996402 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:40.996494 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.996499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.997424 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.997426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.997981 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:40.997988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.998386 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.998388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:40.998462 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:40.998921 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:40.998931 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.000145 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.001923 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.001926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.001928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.002114 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.002355 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.002358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.002360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.002435 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.002776 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.002779 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.002871 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.002951 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.003125 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 07:11:41.003304 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:40.995751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43902 (local address 127.0.98.61:38141)
0924 07:11:40.995994 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:40.995997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:40.996008 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:40.996042 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:40.996044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:40.996082 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:40.996150 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:40.996599 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.997320 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.998065 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:40.998292 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:40.998364 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.000278 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.000282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.000284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.000306 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.001818 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.002196 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.002200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.002202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.002281 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.002508 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.002511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.002513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.002687 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.002776 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.003008 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.003127 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":94,"threads_started":1}
I20260924 07:11:41.004118 7232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:11:41.011610 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.004585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57548)
0924 07:11:41.004690 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.004703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.004774 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.004975 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.004977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.005095 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.005162 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.005166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.006174 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.006177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.006676 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.006683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.007147 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.007150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.007238 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.007737 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.007748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.008904 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.010361 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.010363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.010365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.010551 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.010756 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.010758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.010760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.010834 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.011124 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.011126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.011204 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.011295 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.011461 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 07:11:41.011641 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.004670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.0.98.60:33473)
0924 07:11:41.004824 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.004827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.004837 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:11:41.004863 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.004865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.004895 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.004957 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.005238 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.006070 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.006789 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.007024 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.007107 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.009016 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.009019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.009021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.009040 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.010260 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.010627 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.010630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.010631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.010691 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.010896 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.010898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.010899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.011050 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.011116 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.011327 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.011522 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":91}
I20260924 07:11:41.014634 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Bootstrap starting.
I20260924 07:11:41.017100 7232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:41.017849 7232 log.cc:824] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:41.019898 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: No bootstrap required, opened a new log
I20260924 07:11:41.022940 7232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } }
I20260924 07:11:41.023154 7232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:41.023236 7232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b670153be384e569c195a408e963efe, State: Initialized, Role: FOLLOWER
I20260924 07:11:41.023711 7232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } }
I20260924 07:11:41.024163 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } } }
I20260924 07:11:41.024276 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:41.024930 7232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:11:41.029321 7249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:11:41.029405 7249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:11:41.033224 7080 master_runner.cc:428] Detected that this master 8b670153be384e569c195a408e963efe is joining an existing cluster
I20260924 07:11:41.033361 7080 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33473
I20260924 07:11:41.046846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6576
I20260924 07:11:41.052556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37029
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45079
--webserver_port=42187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:11:41.061924 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.035675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57558)
0924 07:11:41.036277 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.036290 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.036425 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.036735 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.036738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.036932 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.037041 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.037047 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.038046 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.038049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.038930 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.038940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.039613 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.039698 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.040527 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.040541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.056877 (+ 16336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.058724 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.058728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.058730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.058969 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.060869 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.060873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.060950 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.061376 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.061379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.061465 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.061572 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.061721 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"spinlock_wait_cycles":10112,"thread_start_us":99,"threads_started":1}
I20260924 07:11:41.062017 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.036465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.0.98.61:38141)
0924 07:11:41.036582 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.036586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.037043 (+ 457us) server_negotiation.cc:407] Connection header received
0924 07:11:41.037198 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.037201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.037248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.037293 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.037825 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.038627 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039706 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.040003 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.040073 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.056802 (+ 16729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.056808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.056811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.056849 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.059928 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060467 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.060471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.060473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.060569 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.061021 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.061024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.061026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.061262 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.061340 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.061718 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.061854 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:11:41.062523 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.036127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.98.60:33473)
0924 07:11:41.036210 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.036214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.036368 (+ 154us) server_negotiation.cc:407] Connection header received
0924 07:11:41.036513 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.036516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.036550 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.036592 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.037154 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.037916 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039047 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.039405 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039474 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.057000 (+ 17526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.057003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.057006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.057028 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.058594 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060695 (+ 2101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.060699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.060701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.060798 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.061040 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.061043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.061045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.061267 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.061347 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.062268 (+ 921us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.062379 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":29}
I20260924 07:11:41.063292 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.036652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43908)
0924 07:11:41.036954 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.036966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.037100 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.037406 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.037409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.037603 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.037706 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.037713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.038755 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.038757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.039588 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.039595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.040207 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.040211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.040319 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.055972 (+ 15653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.055993 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.056579 (+ 586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.060080 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.060086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.060323 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.060686 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.060691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.060924 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.061427 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.061430 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.061529 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.062792 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.062946 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
I20260924 07:11:41.065531 7255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.035562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47510)
0924 07:11:41.035851 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.035866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.036016 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.036308 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.036310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.036532 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.036757 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.036767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.037820 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.037822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.038739 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.038747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.039439 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.039572 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.040100 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.040113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.057927 (+ 17814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.060791 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.060793 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.060795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.064048 (+ 3253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.064436 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.064439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.064441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.064530 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.065012 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.065014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.065113 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.065210 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.065347 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":17,"thread_start_us":110,"threads_started":1}
I20260924 07:11:41.065555 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.035971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.98.62:45079)
0924 07:11:41.036121 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.036125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.036138 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:41.036179 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.036182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.036229 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.036297 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.036890 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.037665 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.038853 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.039174 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.039245 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.058110 (+ 18865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.058115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.058118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.058149 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.060583 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.064177 (+ 3594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.064181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.064183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.064301 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.064648 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.064652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.064654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.064916 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.065016 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.065284 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.065405 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233899, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:11:41.079492 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.070078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57562)
0924 07:11:41.070194 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.070209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.070304 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.070659 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.070662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.070881 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.070992 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.070999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.072138 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.072141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.073003 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.073014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.073676 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.073680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.073797 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.074308 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.074328 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.075913 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.077862 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.077865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.077868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.078104 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.078381 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.078384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.078386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.078466 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.078920 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.078923 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.079023 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.079117 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.079265 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:11:41.080160 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.070144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.60:33473)
0924 07:11:41.070294 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.070298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.070311 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:41.070415 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.070418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.070471 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.070537 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.071119 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.071999 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.073137 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.073454 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.073542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.076071 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.076075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.076078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.076105 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.077729 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.078204 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.078207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.078210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.078287 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.078554 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.078557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.078559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.078792 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.078878 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.079884 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.080011 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 07:11:41.081156 6514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33473: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8b670153be384e569c195a408e963efe"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33473
}
attrs {
promote: true
}
}
dest_uuid: "f88fc2058d034f66a45c9b616dc7e4a4"
cas_config_opid_index: 5
I20260924 07:11:41.081761 6514 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } }
I20260924 07:11:41.082799 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b670153be384e569c195a408e963efe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.083571 7206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:41.084311 7074 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33473
I20260924 07:11:41.084493 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.084599 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.084933 7206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:11:41.085549 7080 master.cc:537] Master@127.0.98.60:33473 shutting down...
I20260924 07:11:41.085597 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:11:41.087582 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 -> Peer 8b670153be384e569c195a408e963efe (127.0.98.60:33473): Couldn't send request to peer 8b670153be384e569c195a408e963efe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:11:41.091485 7080 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:41.091918 7080 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:41.092120 7080 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: stopping tablet replica
I20260924 07:11:41.093019 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.085360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.98.61:38141)
0924 07:11:41.085490 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.085494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.087872 (+ 2378us) server_negotiation.cc:407] Connection header received
0924 07:11:41.088092 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.088095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.088161 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.088274 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.088594 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.089344 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.091497 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.092190 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.092272 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.092428 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.092801 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.092883 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:11:41.093420 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.082944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:43922)
0924 07:11:41.087786 (+ 4842us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.087804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.087950 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.088310 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.088313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.088332 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.088441 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.088447 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.089478 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.089482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.091299 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.091312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.092527 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.092530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.092628 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.092678 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.093212 (+ 534us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.093295 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4748}
I20260924 07:11:41.093955 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 LEARNER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:11:41.094380 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 07:11:41.096716 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8b670153be384e569c195a408e963efe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.097245 7080 master.cc:552] Master@127.0.98.60:33473 shutdown complete.
I20260924 07:11:41.097316 7080 master_runner.cc:305] Clearing existing system tablet
I20260924 07:11:41.097589 7080 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:41.097679 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:41.098920 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88fc2058d034f66a45c9b616dc7e4a4. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.099045 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:11:41.099134 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: true } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.099213 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:11:41.100013 7073 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: attempting to promote NON_VOTER 08d3f7f2168045999a923578819a9207 to VOTER
I20260924 07:11:41.100466 7073 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } }
I20260924 07:11:41.101838 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.101547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57568)
0924 07:11:41.101695 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.101762 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 07:11:41.101825 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 LEARNER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 07:11:41.102172 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 07:11:41.102692 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.102466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57582)
0924 07:11:41.102590 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.102633 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 07:11:41.102849 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 -> Peer 8b670153be384e569c195a408e963efe (127.0.98.60:33473): Couldn't send request to peer 8b670153be384e569c195a408e963efe. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33473: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:11:41.103881 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 08d3f7f2168045999a923578819a9207 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.105386 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 08d3f7f2168045999a923578819a9207 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.106824 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.106962 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:41.107964 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.108371 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.108258 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.108482 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.109172 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.000s
I20260924 07:11:41.110143 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.110385 7080 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:41.110522 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:41.110554 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "8b670153be384e569c195a408e963efe"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "738347d543938ee45a3e6666238b93f4"
server_key_iv: "faaa64b9051f73c25f7a792c2b4ffbb1"
server_key_version: "encryptionkey@0"
I20260924 07:11:41.112651 7080 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:41.114919 7080 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:11:41.115015 7080 log.cc:1196] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 07:11:41.115242 7080 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Deleting consensus metadata
I20260924 07:11:41.115909 7080 master_runner.cc:315] Copying system tablet from 127.0.98.62:45079
I20260924 07:11:41.117060 7080 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45079
I20260924 07:11:41.126238 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.117703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.98.62:45079)
0924 07:11:41.117827 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.117832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.118079 (+ 247us) server_negotiation.cc:407] Connection header received
0924 07:11:41.118190 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.118194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.118249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.118336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.119150 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.119999 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.121141 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.121480 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.121577 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.122811 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.122816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.122819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.122846 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.124452 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.124891 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.124895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.124897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.124997 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.125254 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.125257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.125259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.125473 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.125561 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.125951 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.126072 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:11:41.127182 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.117605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:47516)
0924 07:11:41.117964 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.117978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.118100 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.118372 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.118375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.118605 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.119034 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.119043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.120111 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.120115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.121014 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.121026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.121597 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.121601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.121693 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.122236 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.122249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.122666 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.124563 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.124567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.124569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.124809 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.125082 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.125084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.125086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.125173 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.125605 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.125608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.125704 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.126720 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.126921 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":80,"threads_started":1}
I20260924 07:11:41.127871 6534 tablet_copy_service.cc:140] P f88fc2058d034f66a45c9b616dc7e4a4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8b670153be384e569c195a408e963efe ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47516)
I20260924 07:11:41.127971 6534 tablet_copy_service.cc:161] P f88fc2058d034f66a45c9b616dc7e4a4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8b670153be384e569c195a408e963efe at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47516: session id = 8b670153be384e569c195a408e963efe-00000000000000000000000000000000
I20260924 07:11:41.129015 6534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:11:41.130322 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:11:41.132601 7080 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: tablet copy: Starting download of 0 data blocks...
I20260924 07:11:41.132822 7080 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: tablet copy: Starting download of 1 WAL segments...
I20260924 07:11:41.134228 7080 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:11:41.136018 6534 tablet_copy_service.cc:342] P f88fc2058d034f66a45c9b616dc7e4a4: Request end of tablet copy session 8b670153be384e569c195a408e963efe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47516
I20260924 07:11:41.136102 6534 tablet_copy_service.cc:433] P f88fc2058d034f66a45c9b616dc7e4a4: ending tablet copy session 8b670153be384e569c195a408e963efe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8b670153be384e569c195a408e963efe
I20260924 07:11:41.143067 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:41.144820 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.144996 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.145171 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:41.145476 7080 server_base.cc:1034] running on GCE node
I20260924 07:11:41.145622 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:41.145948 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:41.147080 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790233901147065 us; error 31 us; skew 500 ppm
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:41.148622 7080 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 07:11:41.149092 7080 webserver.cc:460] Webserver started at http://127.0.98.60:46657/ using document root <none> and password file <none>
I20260924 07:11:41.149281 7080 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:41.149322 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:41.150650 7080 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:41.151644 7281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.151924 7080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:41.151997 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "8b670153be384e569c195a408e963efe"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "738347d543938ee45a3e6666238b93f4"
server_key_iv: "faaa64b9051f73c25f7a792c2b4ffbb1"
server_key_version: "encryptionkey@0"
I20260924 07:11:41.152071 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:41.171738 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:41.174142 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:41.178735 7080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33473
I20260924 07:11:41.178763 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33473 every 8 connection(s)
I20260924 07:11:41.179322 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:11:41.181335 7334 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:11:41.182377 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Bootstrap starting.
I20260924 07:11:41.193430 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:11:41.193745 7334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: Bootstrap complete.
W20260924 07:11:41.193867 7258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:41.194104 7258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:41.194171 7258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:11:41.194377 7334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } }
I20260924 07:11:41.194535 7334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b670153be384e569c195a408e963efe, State: Initialized, Role: LEARNER
I20260924 07:11:41.194643 7334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } }
I20260924 07:11:41.194985 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.195094 7334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:41.195169 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: LEARNER
I20260924 07:11:41.198968 7351 catalog_manager.cc:1259] Loaded cluster ID: e446a3acfb6f46f8b7ee46d434beaab3
I20260924 07:11:41.199083 7351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: loading cluster ID for follower catalog manager: success
W20260924 07:11:41.199421 7258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:41.199479 7258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:41.199571 7258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:41.201537 7351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: acquiring CA information for follower catalog manager: success
I20260924 07:11:41.203011 7351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:11:41.206501 7258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42187
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:41.207644 7258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:41.208743 7258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:41.218402 7353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.218406 7356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:41.218550 7258 server_base.cc:1034] running on GCE node
W20260924 07:11:41.218484 7354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:41.219075 7258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:41.219740 7258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:41.220902 7258 hybrid_clock.cc:648] HybridClock initialized: now 1790233901220882 us; error 31 us; skew 500 ppm
I20260924 07:11:41.221773 6839 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:41.223922 7258 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:11:41.224808 7258 webserver.cc:460] Webserver started at http://127.0.98.1:42187/ using document root <none> and password file <none>
I20260924 07:11:41.225325 7258 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:41.225402 7258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:41.229138 7258 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:41.231185 7363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.232589 7258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:41.232707 7258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "5b3a300362fc43e0ae7cda13d60ba0c3"
format_stamp: "Formatted at 2026-09-24 07:11:39 on dist-test-slave-cnv3"
server_key: "1f1f3ee2e2942f4fb3296a4d404dd9a5"
server_key_iv: "a3e9c38d7c0016de00dbe917d2cc84e9"
server_key_version: "encryptionkey@0"
I20260924 07:11:41.233256 7258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:41.248852 7258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:41.252319 7258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:41.252519 7258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:41.253129 7258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:41.254019 7258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:41.254065 7258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.254108 7258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:41.254136 7258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.263159 7258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37029
I20260924 07:11:41.263195 7476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37029 every 8 connection(s)
I20260924 07:11:41.264307 7258 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:41.269598 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7258
I20260924 07:11:41.269855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6710
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:11:41.278272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35867
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45079
--webserver_port=34233
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:11:41.281172 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.267851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56237 (local address 127.0.98.61:38141)
0924 07:11:41.267973 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.267978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.267992 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:11:41.268525 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.268529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.268593 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.268646 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.270468 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271200 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273547 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.273913 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.274001 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.277027 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.277031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.277033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.277064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.278629 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.279264 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.279269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.279271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.279357 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.279831 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.279835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.279837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.280070 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.280140 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.280890 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.281020 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:11:41.281577 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.266266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43441 (local address 127.0.98.62:45079)
0924 07:11:41.266400 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.266404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.267608 (+ 1204us) server_negotiation.cc:407] Connection header received
0924 07:11:41.268562 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.268566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.268615 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.268679 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.270200 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271026 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273026 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.273375 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.277874 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.277878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.277881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.277908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.279806 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280304 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.280307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.280309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.280414 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280692 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.280695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.280697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.280908 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.280983 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.281299 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.281410 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:11:41.282482 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.266256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33335 (local address 127.0.98.60:33473)
0924 07:11:41.267630 (+ 1374us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.267642 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.267658 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:11:41.268470 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.268474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.268533 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.268613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.270204 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271225 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273026 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.273394 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273487 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.277868 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.277872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.277874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.277904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.279734 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280304 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.280308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.280310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.280543 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280808 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.280811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.280813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.281023 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.281097 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.282218 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.282341 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":85,"threads_started":1}
I20260924 07:11:41.282966 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.267651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.98.1:56237)
0924 07:11:41.267903 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.267916 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.268302 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.268819 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.268827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.269294 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.270322 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.270330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271318 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.271320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.273401 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.273412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.274170 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.274173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.274298 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.274885 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.274900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.276812 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.278765 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.278772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.278792 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.279138 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.279495 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.279498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.279500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.279716 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.280219 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.280226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.280597 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.281407 (+ 810us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.281808 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1448,"spinlock_wait_cycles":8576,"thread_start_us":90,"threads_started":1}
I20260924 07:11:41.283226 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.266639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:43441)
0924 07:11:41.267518 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.267530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.268282 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.268819 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.268827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.269291 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.270047 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.270065 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271182 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.271185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.272883 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.272893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.273667 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.274009 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.274764 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.274793 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.277697 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.279908 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.279910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.279912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.280192 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.280506 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.280511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.280596 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.281036 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.281039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.281127 (+ 88us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.281727 (+ 600us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.281866 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":240,"spinlock_wait_cycles":23808,"thread_start_us":139,"threads_started":1}
I20260924 07:11:41.282966 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.266639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.98.1:33335)
0924 07:11:41.267219 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.267281 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.268260 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.268818 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.268825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.269287 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.270049 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.270066 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.271364 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.271367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.272883 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.272893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.273667 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.273672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.274017 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.274764 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.274794 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.277697 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.279861 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.279865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.279867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.280192 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.280636 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.280639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.280641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.280716 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.281132 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.281135 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.281452 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.281551 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.281998 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":741,"spinlock_wait_cycles":34048,"thread_start_us":646,"threads_started":1}
I20260924 07:11:41.284464 7477 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33473
I20260924 07:11:41.284471 7478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:41.284543 7479 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38141
I20260924 07:11:41.284711 7478 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.284711 7479 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.285274 7478 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
I20260924 07:11:41.285275 7479 heartbeater.cc:511] Master 127.0.98.61:38141 requested a full tablet report, sending...
I20260924 07:11:41.284711 7477 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.285914 7477 heartbeater.cc:511] Master 127.0.98.60:33473 requested a full tablet report, sending...
I20260924 07:11:41.286561 6514 ts_manager.cc:194] Re-registered known tserver with Master: 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1:37029)
I20260924 07:11:41.286924 7298 ts_manager.cc:194] Registered new tserver with Master: 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1:37029)
I20260924 07:11:41.287340 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43441
I20260924 07:11:41.288314 7114 ts_manager.cc:194] Registered new tserver with Master: 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1:37029)
W20260924 07:11:41.392400 7485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:41.392660 7485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:41.392702 7485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:41.396770 7485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:41.396838 7485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:41.396919 7485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:41.402294 7485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34233
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:41.403293 7485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:41.404453 7485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:41.415489 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.415460 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.415905 7493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:41.416347 7485 server_base.cc:1034] running on GCE node
I20260924 07:11:41.416785 7485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:41.417380 7485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:41.418550 7485 hybrid_clock.cc:648] HybridClock initialized: now 1790233901418526 us; error 32 us; skew 500 ppm
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:41.421429 7485 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:11:41.422250 7485 webserver.cc:460] Webserver started at http://127.0.98.2:34233/ using document root <none> and password file <none>
I20260924 07:11:41.422784 7485 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:41.422863 7485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:41.426920 7485 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:41.428996 7500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.430220 7485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:41.430330 7485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "30ce379435aa4f5e8b550c50cd93009c"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "728bc72213f2ba5d5de0d00f4b49d45c"
server_key_iv: "7020effa0b66ab8ae8032f7e4f4d4789"
server_key_version: "encryptionkey@0"
I20260924 07:11:41.430891 7485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:41.434299 6973 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
I20260924 07:11:41.450898 7485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:41.453917 7485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:41.454072 7485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:41.454620 7485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:41.455463 7485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:41.455509 7485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.455549 7485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:41.455569 7485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.463865 7485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35867
I20260924 07:11:41.463892 7613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35867 every 8 connection(s)
I20260924 07:11:41.464838 7485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:41.465436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7485
I20260924 07:11:41.465615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6844
I20260924 07:11:41.472690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45263
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45079
--webserver_port=33605
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:11:41.483407 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.466896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44755 (local address 127.0.98.60:33473)
0924 07:11:41.467027 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.467031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.468634 (+ 1603us) server_negotiation.cc:407] Connection header received
0924 07:11:41.469603 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.469607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.469661 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.469710 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.471629 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472445 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474335 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474684 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474756 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.478683 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.478687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.478689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.478721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.480988 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481547 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.481551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.481553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.481645 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.482059 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.482062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.482064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.482274 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.482345 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.483079 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.483201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:11:41.483418 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.468380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40781 (local address 127.0.98.61:38141)
0924 07:11:41.468508 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.468512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.469110 (+ 598us) server_negotiation.cc:407] Connection header received
0924 07:11:41.469603 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.469607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.469659 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.469710 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.471629 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472450 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474335 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474684 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474756 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.479338 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.479342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.479345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.479395 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.480874 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481547 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.481551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.481553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.481744 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.482059 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.482062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.482064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.482274 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.482342 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.483079 (+ 737us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.483201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:11:41.483430 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.467604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58551 (local address 127.0.98.62:45079)
0924 07:11:41.467782 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.467787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.468558 (+ 771us) server_negotiation.cc:407] Connection header received
0924 07:11:41.469679 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.469683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.469733 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.469796 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.471627 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472358 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474339 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474676 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.479147 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.479151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.479153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.479183 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.480750 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481546 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.481549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.481551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.481647 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.482062 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.482066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.482067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.482301 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.482384 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.483093 (+ 709us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.483219 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 07:11:41.485159 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.467230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:58551)
0924 07:11:41.468407 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.468462 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.469373 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.470029 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.470037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.470679 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.471459 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.471479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472525 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.472528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.474205 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.474216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474982 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.475204 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.476118 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.476133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.478983 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.480894 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.480900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.480919 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481422 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.481755 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.481760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481949 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.482455 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.482458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.482788 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.483586 (+ 798us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.484137 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":345,"spinlock_wait_cycles":101248,"thread_start_us":137,"threads_started":1}
I20260924 07:11:41.485167 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.468277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.98.2:40781)
0924 07:11:41.469011 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.469023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.469374 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.470026 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.470035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.470712 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.471459 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.471479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472584 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.472587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.474194 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.474205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474962 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.475199 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.475977 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.475998 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.479177 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.481000 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.481005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481433 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.481824 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.481829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481956 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.482407 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.482421 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.482783 (+ 362us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.483676 (+ 893us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.484060 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":645,"spinlock_wait_cycles":54016,"thread_start_us":71,"threads_started":1}
I20260924 07:11:41.485167 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.467695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.98.2:44755)
0924 07:11:41.468531 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.468544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.469369 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.470026 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.470035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.470709 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.471464 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.471478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.472584 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.472587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.474206 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.474213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.474962 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.474966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.475211 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.476005 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.476017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.478504 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.481089 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.481094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481443 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.481843 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.481845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.481847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.481955 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.482407 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.482413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.482783 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.484139 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.484254 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":594,"spinlock_wait_cycles":19712,"thread_start_us":116,"threads_started":1}
I20260924 07:11:41.486599 7616 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38141
I20260924 07:11:41.486662 7615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:41.486842 7616 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.486842 7615 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.487460 7616 heartbeater.cc:511] Master 127.0.98.61:38141 requested a full tablet report, sending...
I20260924 07:11:41.487460 7615 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
I20260924 07:11:41.487739 7614 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33473
I20260924 07:11:41.487824 7614 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.487986 7614 heartbeater.cc:511] Master 127.0.98.60:33473 requested a full tablet report, sending...
I20260924 07:11:41.488546 6514 ts_manager.cc:194] Re-registered known tserver with Master: 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:41.488668 7114 ts_manager.cc:194] Registered new tserver with Master: 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:41.489269 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58551
I20260924 07:11:41.489877 7298 ts_manager.cc:194] Registered new tserver with Master: 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:41.496762 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.493653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.98.60:33473)
0924 07:11:41.493753 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.493757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.493769 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:41.493885 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.493888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.493945 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.494017 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.494217 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.494749 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.495808 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.496273 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.496363 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.496498 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.496578 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.496638 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:11:41.496762 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.493599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:57592)
0924 07:11:41.493704 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.493720 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.493800 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.494034 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.494037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.494052 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.494134 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.494138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.494844 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.494848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.495707 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.495717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.496375 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.496379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.496472 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.496492 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.496566 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.496639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:11:41.497298 7308 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 07:11:41.497645 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88fc2058d034f66a45c9b616dc7e4a4. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.497781 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: LEARNER
W20260924 07:11:41.590839 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:41.591120 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:41.591162 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:41.595150 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:41.595204 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:41.595314 7621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:41.600775 7621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33605
--tserver_master_addrs=127.0.98.62:45079,127.0.98.61:38141,127.0.98.60:33473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:41.601871 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:41.603035 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:41.612804 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.612838 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:41.612807 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:41.613133 7621 server_base.cc:1034] running on GCE node
I20260924 07:11:41.613528 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:41.614130 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:41.615307 7621 hybrid_clock.cc:648] HybridClock initialized: now 1790233901615286 us; error 33 us; skew 500 ppm
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:41.618067 7621 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:11:41.618978 7621 webserver.cc:460] Webserver started at http://127.0.98.3:33605/ using document root <none> and password file <none>
I20260924 07:11:41.619534 7621 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:41.619608 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:41.623277 7621 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:41.625386 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.626607 7621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:41.626700 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "8147ec5745404904be9e1c1f9ddc2de7"
format_stamp: "Formatted at 2026-09-24 07:11:40 on dist-test-slave-cnv3"
server_key: "f63b4a1ad0fead4fcf9f88f4c830bb66"
server_key_iv: "11e59667d001a1ff2b39565be9ab3603"
server_key_version: "encryptionkey@0"
I20260924 07:11:41.627252 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:41.646443 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:41.649729 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:41.649904 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:41.650487 7621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:41.651337 7621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:41.651412 7621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.651484 7621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:41.651512 7621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:41.660342 7621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45263
I20260924 07:11:41.660405 7749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45263 every 8 connection(s)
I20260924 07:11:41.661427 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:41.669855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7621
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 07:11:41 dist-test-slave-cnv3 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233901, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:11:41.683334 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.672574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.0.98.60:33473)
0924 07:11:41.673083 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.673094 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.673586 (+ 492us) server_negotiation.cc:407] Connection header received
0924 07:11:41.673633 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.673636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.673696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.673775 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.675887 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.676674 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.677828 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.678168 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.678264 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.679864 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.679869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.679871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.679898 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.681406 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.681837 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.681840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.681842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.681919 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.682186 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.682189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.682191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.682385 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.682494 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.683043 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.683159 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":102,"threads_started":1}
I20260924 07:11:41.683848 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.665057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.98.3:55135)
0924 07:11:41.665281 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.665294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.665546 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.666099 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.666106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.666634 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.667265 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.667282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.668111 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.668114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.669470 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.669480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.672176 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.672180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.672265 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.672692 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.672724 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.674641 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.677108 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.677116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.677136 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.677509 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.680489 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.680492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.680495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.680723 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.681617 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.681625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.682014 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.682229 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.682617 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":32,"spinlock_wait_cycles":8064,"thread_start_us":74,"threads_started":1}
I20260924 07:11:41.684379 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.673075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.98.62:45079)
0924 07:11:41.674836 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.674851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.674862 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:41.674896 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.674899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.674952 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.675029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.676715 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.677532 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.678693 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.679045 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.679124 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.680598 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.680602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.680604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.680629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.682238 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.682657 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.682660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.682663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.682737 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.683102 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.683105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.683107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.683322 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.683463 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.684102 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.684223 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":90,"threads_started":1}
I20260924 07:11:41.685756 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.665186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55135 (local address 127.0.98.61:38141)
0924 07:11:41.665306 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.665309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.665352 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:11:41.665776 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.665811 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.665859 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.665910 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.667450 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.667968 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.671685 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.672023 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.672135 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.674804 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.674808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.674811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.674841 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.676978 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.680254 (+ 3276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.680259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.680261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.680374 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.681228 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.681232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.681234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.681474 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.681551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.685480 (+ 3929us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.685609 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:41.686259 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.663855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.98.3:60021)
0924 07:11:41.664819 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.664831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.665525 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.666239 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.666242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.666656 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.667273 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.667282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.669005 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.669007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.669622 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.669632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.670222 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.670226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.670887 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.671764 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.671795 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.674367 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.678492 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.678495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.678497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.678952 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.681032 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.681035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.681037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.681118 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.685418 (+ 4300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.685421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.685511 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.685605 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.685739 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":115,"spinlock_wait_cycles":82048,"thread_start_us":679,"threads_started":1}
I20260924 07:11:41.687067 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.663292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60021 (local address 127.0.98.60:33473)
0924 07:11:41.663460 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.663464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.664904 (+ 1440us) server_negotiation.cc:407] Connection header received
0924 07:11:41.666052 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.666055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.666112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.666198 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.668112 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.668881 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.669762 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.670063 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.670163 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.674576 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.674580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.674583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.674611 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.678385 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.680839 (+ 2454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.680842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.680844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.680929 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.685121 (+ 4192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.685123 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.685125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.685299 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.685376 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.686815 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.686933 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:11:41.687328 7752 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38141
I20260924 07:11:41.687579 7752 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.688050 7752 heartbeater.cc:511] Master 127.0.98.61:38141 requested a full tablet report, sending...
I20260924 07:11:41.689319 7114 ts_manager.cc:194] Registered new tserver with Master: 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
I20260924 07:11:41.689738 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.663548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56797 (local address 127.0.98.62:45079)
0924 07:11:41.663665 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.663668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.664570 (+ 902us) server_negotiation.cc:407] Connection header received
0924 07:11:41.665776 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.665779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.665841 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.665923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.669057 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.669781 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.670827 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.671134 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.671215 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.675352 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.675356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.675359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.675417 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.680143 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.688497 (+ 8354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.688501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.688503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.688589 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.688859 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.688862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.688864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.689059 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.689139 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.689483 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.689599 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:11:41.690194 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.663855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:56797)
0924 07:11:41.664399 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.664449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.665525 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.666097 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.666105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.666656 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:41.667277 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.667282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.669886 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.669889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.670668 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.670679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.671410 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.671414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.671570 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.672050 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:41.672065 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:41.674759 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:41.688107 (+ 13348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.688110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.688112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.688368 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.688690 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:41.688693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:41.688694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:41.688770 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:41.689194 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:41.689197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:41.689299 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.689861 (+ 562us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.690019 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":70,"spinlock_wait_cycles":49024,"thread_start_us":163,"threads_started":1}
I20260924 07:11:41.690599 7750 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33473
I20260924 07:11:41.690685 7750 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.687099 6514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:11:41.690853 7750 heartbeater.cc:511] Master 127.0.98.60:33473 requested a full tablet report, sending...
I20260924 07:11:41.691156 7751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45079
I20260924 07:11:41.691232 7751 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:41.691434 7751 heartbeater.cc:511] Master 127.0.98.62:45079 requested a full tablet report, sending...
I20260924 07:11:41.691558 7298 ts_manager.cc:194] Registered new tserver with Master: 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
I20260924 07:11:41.692081 6513 ts_manager.cc:194] Re-registered known tserver with Master: 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
W20260924 07:11:41.692492 6514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:41.692834 6513 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56797
I20260924 07:11:41.694931 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.673789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49374 (local address 127.0.98.61:38141)
0924 07:11:41.674309 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.674322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.674593 (+ 271us) server_negotiation.cc:407] Connection header received
0924 07:11:41.674880 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.674883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.674931 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.675011 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.687116 (+ 12105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.687936 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.689059 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.689356 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.689817 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.691039 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.691043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.691046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.691070 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.692752 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.693227 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.693230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.693231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.693298 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.693683 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.693686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.693687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.693897 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.694531 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.694603 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.694722 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":63,"threads_started":1}
I20260924 07:11:41.695143 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: true } } }
I20260924 07:11:41.695258 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.698762 7263 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: attempting to promote NON_VOTER 8b670153be384e569c195a408e963efe to VOTER
I20260924 07:11:41.699057 7263 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:41.700134 7124 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:11:41.700130 7308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 LEARNER]: Refusing update from remote peer f88fc2058d034f66a45c9b616dc7e4a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:11:41.700551 7768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:11:41.700825 7768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:11:41.702392 7263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b670153be384e569c195a408e963efe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.702832 7124 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b670153be384e569c195a408e963efe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.703754 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.703858 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.703976 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.704051 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.704164 7263 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.704242 7263 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:41.704001 7308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8b670153be384e569c195a408e963efe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.705535 7340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.705646 7340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:41.706054 7777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:41.706187 7777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:41.713690 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.708679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37029 (local address 127.0.0.1:44324)
0924 07:11:41.708843 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.708859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.708936 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.709742 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.709745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.709762 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.709870 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.709876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.711025 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.711029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.712129 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.712139 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.713319 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.713323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.713409 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.713429 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.713487 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.713557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 07:11:41.713732 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.708861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.0.98.1:37029)
0924 07:11:41.709163 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.709174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.709198 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:11:41.709281 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.709287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.709583 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.710051 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.710188 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.710908 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.712267 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.713210 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.713303 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.713472 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.713521 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.713609 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
I20260924 07:11:41.715616 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.709256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45263 (local address 127.0.0.1:50320)
0924 07:11:41.709798 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.709819 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.709892 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.711674 (+ 1782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.711677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.711697 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.711791 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.711797 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.713057 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.713061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.714276 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.714286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.715179 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.715182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.715267 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.715289 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.715362 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.715472 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":335,"thread_start_us":127,"threads_started":1}
I20260924 07:11:41.715634 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.709546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.3:45263)
0924 07:11:41.711184 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.711195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.711217 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:11:41.711290 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.711296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.711532 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.711683 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.711899 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.712963 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.714401 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.715064 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.715161 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.715298 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.715424 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.715511 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":100,"threads_started":1}
I20260924 07:11:41.715953 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.709256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.0.1:44768)
0924 07:11:41.709510 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.709522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.709599 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.712012 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.712014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.712032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.712110 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.712116 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.713514 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.713517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.714571 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.714578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.715590 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.715594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.715680 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.715702 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.715745 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.715822 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260924 07:11:41.716533 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.709886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.0.98.2:35867)
0924 07:11:41.711588 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.711596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.711615 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:11:41.711687 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.711692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.711890 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.712030 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.712347 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.713390 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.714683 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.715452 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.716128 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.716263 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.716311 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.716399 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1616,"thread_start_us":89,"threads_started":1}
I20260924 07:11:41.717226 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 02a9748964804768a0ba0f2c32ff0258 (DEFAULT_TABLE table=test-table [id=97ca5dd12d08435fbf9881097af3fb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:41.718158 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a9748964804768a0ba0f2c32ff0258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:41.721956 7548 tablet_service.cc:1467] Processing CreateTablet for tablet 02a9748964804768a0ba0f2c32ff0258 (DEFAULT_TABLE table=test-table [id=97ca5dd12d08435fbf9881097af3fb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:41.721953 7684 tablet_service.cc:1467] Processing CreateTablet for tablet 02a9748964804768a0ba0f2c32ff0258 (DEFAULT_TABLE table=test-table [id=97ca5dd12d08435fbf9881097af3fb9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:41.722872 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a9748964804768a0ba0f2c32ff0258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:41.722916 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a9748964804768a0ba0f2c32ff0258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:41.728942 7783 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Bootstrap starting.
I20260924 07:11:41.730183 7785 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Bootstrap starting.
I20260924 07:11:41.730588 7784 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Bootstrap starting.
I20260924 07:11:41.731346 7783 tablet_bootstrap.cc:654] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:41.732478 7783 log.cc:824] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:41.732971 7785 tablet_bootstrap.cc:654] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:41.733206 7784 tablet_bootstrap.cc:654] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:41.733732 7785 log.cc:824] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:41.733856 7784 log.cc:824] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:41.735127 7783 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: No bootstrap required, opened a new log
I20260924 07:11:41.735321 7783 ts_tablet_manager.cc:1397] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:11:41.735652 7785 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: No bootstrap required, opened a new log
I20260924 07:11:41.735827 7785 ts_tablet_manager.cc:1397] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:11:41.735973 7784 tablet_bootstrap.cc:492] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: No bootstrap required, opened a new log
I20260924 07:11:41.736145 7784 ts_tablet_manager.cc:1397] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:11:41.739128 7785 raft_consensus.cc:348] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.739339 7785 raft_consensus.cc:374] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:41.739167 7783 raft_consensus.cc:348] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.739408 7785 raft_consensus.cc:729] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ce379435aa4f5e8b550c50cd93009c, State: Initialized, Role: FOLLOWER
I20260924 07:11:41.739426 7783 raft_consensus.cc:374] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:41.739475 7783 raft_consensus.cc:729] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8147ec5745404904be9e1c1f9ddc2de7, State: Initialized, Role: FOLLOWER
I20260924 07:11:41.739822 7785 consensus_queue.cc:260] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.739830 7784 raft_consensus.cc:348] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.739929 7783 consensus_queue.cc:260] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.740046 7784 raft_consensus.cc:374] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:41.740087 7784 raft_consensus.cc:729] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3a300362fc43e0ae7cda13d60ba0c3, State: Initialized, Role: FOLLOWER
I20260924 07:11:41.740469 7784 consensus_queue.cc:260] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.740777 7785 ts_tablet_manager.cc:1428] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:11:41.740823 7783 ts_tablet_manager.cc:1428] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 07:11:41.741262 7784 ts_tablet_manager.cc:1428] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:11:41.741935 7615 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
I20260924 07:11:41.742252 7478 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
I20260924 07:11:41.743711 7751 heartbeater.cc:503] Master 127.0.98.62:45079 was elected leader, sending a full tablet report...
W20260924 07:11:41.766434 7480 tablet.cc:2369] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:41.789721 7791 raft_consensus.cc:482] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:41.789922 7791 raft_consensus.cc:504] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.791092 7791 leader_election.cc:288] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867), 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
I20260924 07:11:41.794833 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.791487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48343 (local address 127.0.98.2:35867)
0924 07:11:41.791636 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.791641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.791880 (+ 239us) server_negotiation.cc:407] Connection header received
0924 07:11:41.791994 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.791996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.792046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.792120 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.792318 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.792813 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.793920 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.794295 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.794385 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.794448 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.794626 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.794698 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:11:41.794850 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.791359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45263 (local address 127.0.98.1:51521)
0924 07:11:41.791706 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.791718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.791817 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.792058 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.792060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.792077 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.792183 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.792191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.792938 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.792941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.793809 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.793819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.794405 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.794409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.794546 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.794564 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.794633 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.794705 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":68,"thread_start_us":98,"threads_started":1}
I20260924 07:11:41.794863 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.791480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51521 (local address 127.0.98.3:45263)
0924 07:11:41.791642 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.791647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.791788 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:11:41.791912 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.791916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.791969 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.792032 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:41.792285 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.792832 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.793924 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.794295 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.794385 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.794458 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.794685 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.794753 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:11:41.794857 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.791366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.98.1:48343)
0924 07:11:41.791789 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:41.791803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:41.791902 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:41.792139 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:41.792142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:41.792158 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:41.792226 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.792230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.792915 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.792918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.793804 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:41.793813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.794406 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.794409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:41.794513 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.794541 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 07:11:41.794600 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:11:41.794713 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":48,"thread_start_us":86,"threads_started":1}
I20260924 07:11:41.795578 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a9748964804768a0ba0f2c32ff0258" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ce379435aa4f5e8b550c50cd93009c" is_pre_election: true
I20260924 07:11:41.795830 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a9748964804768a0ba0f2c32ff0258" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8147ec5745404904be9e1c1f9ddc2de7" is_pre_election: true
I20260924 07:11:41.795925 7568 raft_consensus.cc:2454] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 0.
I20260924 07:11:41.796121 7704 raft_consensus.cc:2454] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 0.
I20260924 07:11:41.796443 7367 leader_election.cc:302] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ce379435aa4f5e8b550c50cd93009c, 5b3a300362fc43e0ae7cda13d60ba0c3; no voters:
I20260924 07:11:41.796772 7791 raft_consensus.cc:2781] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:41.796860 7791 raft_consensus.cc:482] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:41.796893 7791 raft_consensus.cc:3037] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:41.797883 7791 raft_consensus.cc:504] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.798233 7791 leader_election.cc:288] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867), 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263)
I20260924 07:11:41.798552 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a9748964804768a0ba0f2c32ff0258" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ce379435aa4f5e8b550c50cd93009c"
I20260924 07:11:41.798573 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a9748964804768a0ba0f2c32ff0258" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8147ec5745404904be9e1c1f9ddc2de7"
I20260924 07:11:41.798702 7704 raft_consensus.cc:3037] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:41.798702 7568 raft_consensus.cc:3037] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:41.800107 7568 raft_consensus.cc:2454] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 1.
I20260924 07:11:41.800397 7367 leader_election.cc:302] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ce379435aa4f5e8b550c50cd93009c, 5b3a300362fc43e0ae7cda13d60ba0c3; no voters:
I20260924 07:11:41.800588 7791 raft_consensus.cc:2781] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:41.800596 7704 raft_consensus.cc:2454] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 1.
I20260924 07:11:41.800798 7791 raft_consensus.cc:686] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 LEADER]: Becoming Leader. State: Replica: 5b3a300362fc43e0ae7cda13d60ba0c3, State: Running, Role: LEADER
I20260924 07:11:41.801141 7791 consensus_queue.cc:237] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } }
I20260924 07:11:41.804579 6514 catalog_manager.cc:5495] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:41.824200 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.820701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.98.1:37029)
0924 07:11:41.820830 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.820833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.820848 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:41.820923 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.820925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.820982 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.821072 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:41.821328 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.821849 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.822722 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.822959 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.823039 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.823362 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:41.823863 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:41.823981 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.824030 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.824086 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:11:41.830304 7704 raft_consensus.cc:1261] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Refusing update from remote peer 5b3a300362fc43e0ae7cda13d60ba0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:41.830332 7568 raft_consensus.cc:1261] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Refusing update from remote peer 5b3a300362fc43e0ae7cda13d60ba0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:41.831281 7794 consensus_queue.cc:1035] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:41.831563 7791 consensus_queue.cc:1035] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:41.842567 7798 mvcc.cc:205] Tried to move back new op lower bound from 7332798061888954368 to 7332798061780160512. Current Snapshot: MvccSnapshot[applied={T|T < 7332798061888954368}]
I20260924 07:11:41.845417 7799 mvcc.cc:205] Tried to move back new op lower bound from 7332798061888954368 to 7332798061780160512. Current Snapshot: MvccSnapshot[applied={T|T < 7332798061888954368}]
I20260924 07:11:41.846681 7797 mvcc.cc:205] Tried to move back new op lower bound from 7332798061888954368 to 7332798061780160512. Current Snapshot: MvccSnapshot[applied={T|T < 7332798061888954368}]
I20260924 07:11:41.855345 6513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:41.855542 6513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:41.859822 6513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f88fc2058d034f66a45c9b616dc7e4a4: Sending DeleteTablet for 3 replicas of tablet 02a9748964804768a0ba0f2c32ff0258
I20260924 07:11:41.860791 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a9748964804768a0ba0f2c32ff0258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44324
I20260924 07:11:41.861075 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a9748964804768a0ba0f2c32ff0258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44768
I20260924 07:11:41.861352 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a9748964804768a0ba0f2c32ff0258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50320
I20260924 07:11:41.861421 7807 tablet_replica.cc:330] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: stopping tablet replica
I20260924 07:11:41.861830 7807 raft_consensus.cc:2229] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:41.861922 7809 tablet_replica.cc:330] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: stopping tablet replica
I20260924 07:11:41.862228 7809 raft_consensus.cc:2229] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:41.862427 7807 raft_consensus.cc:2258] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:41.862537 7809 raft_consensus.cc:2258] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:41.862885 7808 tablet_replica.cc:330] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: stopping tablet replica
I20260924 07:11:41.863165 7808 raft_consensus.cc:2229] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:41.863489 7808 raft_consensus.cc:2258] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:41.864405 7807 ts_tablet_manager.cc:1905] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:41.864449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6482
I20260924 07:11:41.866204 7808 ts_tablet_manager.cc:1905] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:41.869022 7807 ts_tablet_manager.cc:1918] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:41.871809 7808 ts_tablet_manager.cc:1918] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:41.872000 7807 log.cc:1196] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/02a9748964804768a0ba0f2c32ff0258
I20260924 07:11:41.872406 7807 ts_tablet_manager.cc:1939] T 02a9748964804768a0ba0f2c32ff0258 P 5b3a300362fc43e0ae7cda13d60ba0c3: Deleting consensus metadata
I20260924 07:11:41.872589 7808 log.cc:1196] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/02a9748964804768a0ba0f2c32ff0258
I20260924 07:11:41.872974 7808 ts_tablet_manager.cc:1939] T 02a9748964804768a0ba0f2c32ff0258 P 30ce379435aa4f5e8b550c50cd93009c: Deleting consensus metadata
I20260924 07:11:41.873898 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.863479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49382 (local address 127.0.98.61:38141)
0924 07:11:41.863596 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.863600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.864142 (+ 542us) server_negotiation.cc:407] Connection header received
0924 07:11:41.864317 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.864320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.864368 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.864415 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.865989 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.866800 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.868041 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.868397 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.868466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.870407 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.870411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.870413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.870442 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.872084 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.872575 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.872579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.872581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.872664 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.872936 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.872939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.872941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.873164 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.873260 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.873602 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.873730 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 07:11:41.875918 7502 connection.cc:441] server connection from 127.0.0.1:44768 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44768 (request call id 1) could send its response
W20260924 07:11:41.876359 7365 connection.cc:441] server connection from 127.0.0.1:44324 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44324 (request call id 1) could send its response
I20260924 07:11:41.877378 7809 ts_tablet_manager.cc:1905] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:41.880115 7809 ts_tablet_manager.cc:1918] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:41.880210 7809 log.cc:1196] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/02a9748964804768a0ba0f2c32ff0258
I20260924 07:11:41.880553 7809 ts_tablet_manager.cc:1939] T 02a9748964804768a0ba0f2c32ff0258 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting consensus metadata
W20260924 07:11:41.881651 7638 connection.cc:441] server connection from 127.0.0.1:50320 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50320 (request call id 1) could send its response
I20260924 07:11:41.883308 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.863595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.98.60:33473)
0924 07:11:41.863698 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.863701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.874795 (+ 11094us) server_negotiation.cc:407] Connection header received
0924 07:11:41.874954 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.874958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.875016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.875064 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.875893 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.876688 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.877800 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.878099 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.878198 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.879853 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.879857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.879859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.879886 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.881646 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.882180 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.882184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.882186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.882265 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.882526 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.882529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.882531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.882699 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.882814 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.883073 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.883180 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:11:41.887104 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.878378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.0.98.61:38141)
0924 07:11:41.878475 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.878479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.879125 (+ 646us) server_negotiation.cc:407] Connection header received
0924 07:11:41.879275 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.879278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.879317 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.879361 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.880160 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.880926 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.882090 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.882414 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.882478 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.883949 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.883953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.883955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.883978 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.885576 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.885920 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.885923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.885925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.885999 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.886228 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.886231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.886232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.886421 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.886549 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.886846 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.886952 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:11:41.891836 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:41.884206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.0.98.60:33473)
0924 07:11:41.884307 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:41.884310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:41.884621 (+ 311us) server_negotiation.cc:407] Connection header received
0924 07:11:41.884701 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:41.884704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:41.884749 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:41.884818 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:41.885546 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.886318 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.887407 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:41.887715 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:41.887786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:41.889121 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:41.889125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:41.889128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:41.889151 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:41.890250 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.890651 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.890654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.890656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.890730 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:41.890958 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:41.890968 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:41.890970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:41.891131 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:41.891238 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:41.891588 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 07:11:41.891691 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:11:42.845929 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:42.845372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:36623)
0924 07:11:42.845707 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:42.845810 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":78,"threads_started":1}
W20260924 07:11:42.846266 7478 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:42.847363 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:42.846622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:56377)
0924 07:11:42.847175 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:42.847266 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":64,"threads_started":1}
I20260924 07:11:42.847586 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:42.847019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:48523)
0924 07:11:42.847363 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:42.847501 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":58,"threads_started":1}
W20260924 07:11:42.847710 7615 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
W20260924 07:11:42.847812 7751 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45079 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:43.360739 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f88fc2058d034f66a45c9b616dc7e4a4)
I20260924 07:11:43.360948 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:43.361023 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f88fc2058d034f66a45c9b616dc7e4a4)
I20260924 07:11:43.361197 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:43.361928 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 08d3f7f2168045999a923578819a9207 (127.0.98.61:38141)
I20260924 07:11:43.362039 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 8b670153be384e569c195a408e963efe (127.0.98.60:33473)
I20260924 07:11:43.362747 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42246)
0924 07:11:43.362558 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.362646 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":67,"threads_started":1}
I20260924 07:11:43.362782 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42244)
0924 07:11:43.362611 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.362690 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":60,"threads_started":1}
W20260924 07:11:43.362941 7099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
W20260924 07:11:43.362941 7283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:43.363890 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.363655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42270)
0924 07:11:43.363781 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.363827 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 07:11:43.363890 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.363664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42256)
0924 07:11:43.363784 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.363827 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 07:11:43.364017 7283 leader_election.cc:334] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
W20260924 07:11:43.364038 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:43.366294 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33473 (local address 127.0.0.1:41160)
0924 07:11:43.362667 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.362678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:43.362768 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:43.363138 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:43.363142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:43.363163 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:43.363278 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:43.363285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.364382 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.364386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:43.365214 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:43.365223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.365873 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.365877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:43.365975 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:43.366001 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:11:43.366085 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:11:43.366169 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":51,"threads_started":1}
I20260924 07:11:43.366305 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.60:33473)
0924 07:11:43.362914 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:43.362918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:43.362930 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:43.362971 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:43.362974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:43.363014 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:43.363408 (+ 394us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:43.363585 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.364286 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.365331 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.365764 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.365840 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:43.365888 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:43.366110 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 07:11:43.366179 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":464,"thread_start_us":75,"threads_started":1}
I20260924 07:11:43.366480 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38141 (local address 127.0.0.1:49394)
0924 07:11:43.362619 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.362631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:43.362743 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:43.363346 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:43.363348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:43.363368 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:43.363552 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:43.363560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.364577 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.364579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:43.365392 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:43.365401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.366119 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.366122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:43.366216 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:43.366237 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:11:43.366309 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:11:43.366374 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260924 07:11:43.366824 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.362489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.61:38141)
0924 07:11:43.363028 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:43.363032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:43.363043 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:43.363080 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:43.363083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:43.363133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:43.363208 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:43.363688 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.364483 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.365525 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:43.366023 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:43.366107 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:43.366155 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:43.366334 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 07:11:43.366737 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":42,"threads_started":1}
I20260924 07:11:43.366902 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08d3f7f2168045999a923578819a9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b670153be384e569c195a408e963efe" is_pre_election: true
I20260924 07:11:43.367125 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d3f7f2168045999a923578819a9207 in term 1.
I20260924 07:11:43.367182 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b670153be384e569c195a408e963efe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08d3f7f2168045999a923578819a9207" is_pre_election: true
I20260924 07:11:43.367411 7124 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b670153be384e569c195a408e963efe in term 1.
I20260924 07:11:43.367498 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d3f7f2168045999a923578819a9207, 8b670153be384e569c195a408e963efe; no voters: f88fc2058d034f66a45c9b616dc7e4a4
I20260924 07:11:43.367734 7821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:11:43.367798 7821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f88fc2058d034f66a45c9b616dc7e4a4)
I20260924 07:11:43.367771 7283 leader_election.cc:302] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d3f7f2168045999a923578819a9207, 8b670153be384e569c195a408e963efe; no voters: f88fc2058d034f66a45c9b616dc7e4a4
I20260924 07:11:43.367842 7821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:11:43.367967 7820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:11:43.368021 7820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Starting leader election (detected failure of leader f88fc2058d034f66a45c9b616dc7e4a4)
I20260924 07:11:43.368065 7820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:11:43.368876 7821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:43.368960 7820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:43.369247 7821 leader_election.cc:288] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 election: Requested vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 8b670153be384e569c195a408e963efe (127.0.98.60:33473)
I20260924 07:11:43.369295 7820 leader_election.cc:288] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 election: Requested vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 08d3f7f2168045999a923578819a9207 (127.0.98.61:38141)
I20260924 07:11:43.369590 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.369367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42274)
0924 07:11:43.369469 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.369525 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:11:43.369570 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08d3f7f2168045999a923578819a9207" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b670153be384e569c195a408e963efe"
I20260924 07:11:43.369771 7308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 08d3f7f2168045999a923578819a9207 in current term 2: Already voted for candidate 8b670153be384e569c195a408e963efe in this term.
I20260924 07:11:43.369956 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b670153be384e569c195a408e963efe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08d3f7f2168045999a923578819a9207"
I20260924 07:11:43.370105 7124 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b670153be384e569c195a408e963efe in current term 2: Already voted for candidate 08d3f7f2168045999a923578819a9207 in this term.
I20260924 07:11:43.370543 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.370358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42300)
0924 07:11:43.370453 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.370495 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:11:43.370587 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.369609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42286)
0924 07:11:43.370479 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.370535 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811}
W20260924 07:11:43.370652 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:43.370718 7099 leader_election.cc:302] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08d3f7f2168045999a923578819a9207; no voters: 8b670153be384e569c195a408e963efe, f88fc2058d034f66a45c9b616dc7e4a4
I20260924 07:11:43.370967 7821 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:11:43.371241 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.371037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42316)
0924 07:11:43.371142 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.371193 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 07:11:43.371330 7283 leader_election.cc:334] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:43.371423 7283 leader_election.cc:302] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b670153be384e569c195a408e963efe; no voters: 08d3f7f2168045999a923578819a9207, f88fc2058d034f66a45c9b616dc7e4a4
I20260924 07:11:43.371542 7820 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:11:43.847489 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.846938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:33329)
0924 07:11:43.847268 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.847348 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260924 07:11:43.848592 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.848193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:55769)
0924 07:11:43.848444 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.848499 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260924 07:11:43.848779 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:43.848355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:44777)
0924 07:11:43.848610 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:43.848670 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":58,"threads_started":1}
I20260924 07:11:44.848973 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:44.848401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:55995)
0924 07:11:44.848714 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:44.848860 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":76,"threads_started":1}
I20260924 07:11:44.849810 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:44.849351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:41633)
0924 07:11:44.849639 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:44.849698 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":67,"threads_started":1}
I20260924 07:11:44.850109 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:44.849529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:60367)
0924 07:11:44.849966 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:44.850021 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":60,"threads_started":1}
I20260924 07:11:45.019467 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:45.019593 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:45.020007 7836 leader_election.cc:288] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 8b670153be384e569c195a408e963efe (127.0.98.60:33473)
I20260924 07:11:45.020394 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08d3f7f2168045999a923578819a9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b670153be384e569c195a408e963efe" is_pre_election: true
I20260924 07:11:45.020501 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.020136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42580)
0924 07:11:45.020353 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.020406 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260924 07:11:45.020546 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d3f7f2168045999a923578819a9207 in term 2.
I20260924 07:11:45.020838 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d3f7f2168045999a923578819a9207, 8b670153be384e569c195a408e963efe; no voters:
I20260924 07:11:45.021010 7836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 07:11:45.021073 7836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:45.021119 7836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:11:45.021346 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.021125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42582)
0924 07:11:45.021230 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.021283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:11:45.021454 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:45.022212 7836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:45.022529 7836 leader_election.cc:288] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 election: Requested vote from peers f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079), 8b670153be384e569c195a408e963efe (127.0.98.60:33473)
I20260924 07:11:45.022894 7308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08d3f7f2168045999a923578819a9207" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8b670153be384e569c195a408e963efe"
I20260924 07:11:45.022984 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.022715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42596)
0924 07:11:45.022870 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.022923 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:11:45.023027 7308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:11:45.023612 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.023418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42612)
0924 07:11:45.023507 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.023554 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 07:11:45.023730 7099 leader_election.cc:334] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
I20260924 07:11:45.024183 7308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d3f7f2168045999a923578819a9207 in term 3.
I20260924 07:11:45.024461 7100 leader_election.cc:302] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08d3f7f2168045999a923578819a9207, 8b670153be384e569c195a408e963efe; no voters: f88fc2058d034f66a45c9b616dc7e4a4
I20260924 07:11:45.024637 7836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:11:45.024797 7836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [term 3 LEADER]: Becoming Leader. State: Replica: 08d3f7f2168045999a923578819a9207, State: Running, Role: LEADER
I20260924 07:11:45.025038 7836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } }
I20260924 07:11:45.026046 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d3f7f2168045999a923578819a9207. Latest consensus state: current_term: 3 leader_uuid: "08d3f7f2168045999a923578819a9207" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:45.026139 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:45.026418 7841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:45.028682 7841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:45.029078 7841 catalog_manager.cc:1259] Loaded cluster ID: e446a3acfb6f46f8b7ee46d434beaab3
I20260924 07:11:45.029121 7841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:45.029673 7841 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:45.030045 7841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Loaded TSK: 0
I20260924 07:11:45.030750 7841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:11:45.104579 7113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:45.107741 7113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:45.110901 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.110612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42630)
0924 07:11:45.110735 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.110800 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:11:45.111068 7308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [term 3 FOLLOWER]: Refusing update from remote peer 08d3f7f2168045999a923578819a9207: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 07:11:45.111620 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.111414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.0.1:42646)
0924 07:11:45.111503 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.111555 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:11:45.111582 7839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 07:11:45.111778 7099 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 -> Peer f88fc2058d034f66a45c9b616dc7e4a4 (127.0.98.62:45079): Couldn't send request to peer f88fc2058d034f66a45c9b616dc7e4a4. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:11:45.113138 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d3f7f2168045999a923578819a9207. Latest consensus state: current_term: 3 leader_uuid: "08d3f7f2168045999a923578819a9207" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:45.113265 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:45.114262 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "08d3f7f2168045999a923578819a9207" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:45.114393 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:45.114667 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "08d3f7f2168045999a923578819a9207" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f88fc2058d034f66a45c9b616dc7e4a4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45079 } } peers { permanent_uuid: "08d3f7f2168045999a923578819a9207" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38141 } attrs { promote: false } } peers { permanent_uuid: "8b670153be384e569c195a408e963efe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33473 } attrs { promote: false } } }
I20260924 07:11:45.114784 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b670153be384e569c195a408e963efe [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:11:45.123668 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.119687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45263 (local address 127.0.0.1:50326)
0924 07:11:45.119793 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.119807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:45.119902 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:45.120320 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:45.120323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:45.120340 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:45.120447 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.120453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.121496 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.121499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.122382 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.122390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.123257 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.123260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.123403 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.123432 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:11:45.123481 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:11:45.123556 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260924 07:11:45.124146 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.119860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.0.98.3:45263)
0924 07:11:45.120093 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:45.120097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:45.120110 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:45.120159 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:45.120162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:45.120207 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:45.120550 (+ 343us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:45.120655 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.121403 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.122511 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.123131 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.123850 (+ 719us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.123915 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:45.123944 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 07:11:45.124010 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260924 07:11:45.124399 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.119943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37029 (local address 127.0.0.1:44334)
0924 07:11:45.120598 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.120612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:45.120692 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:45.121090 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:45.121093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:45.121105 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:45.121178 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.121182 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.122159 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.122163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.123166 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.123174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.124031 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.124034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.124117 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.124139 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:45.124211 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:11:45.124280 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":26,"thread_start_us":100,"threads_started":1}
I20260924 07:11:45.124454 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.120427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44334 (local address 127.0.98.1:37029)
0924 07:11:45.120865 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:45.120869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:45.120881 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:45.120929 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:45.120932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:45.120974 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:45.121083 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:45.121289 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.122043 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.123297 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.123916 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.124014 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.124074 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:45.124229 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:11:45.124299 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":68,"threads_started":1}
I20260924 07:11:45.124804 7684 tablet_service.cc:1467] Processing CreateTablet for tablet 323a417ca3304694942fcae02a344d50 (DEFAULT_TABLE table=test-table [id=9b881f017db94461bf61b0fe6cb47a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:45.125023 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.120224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35867 (local address 127.0.0.1:44780)
0924 07:11:45.120864 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.120876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:45.120963 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:45.121283 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:45.121286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:45.121302 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:45.121380 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.121385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.122328 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.122331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.123468 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:45.123475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.124662 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.124665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:45.124749 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.124771 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:45.124836 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:11:45.124902 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":93,"threads_started":1}
I20260924 07:11:45.125083 7684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a417ca3304694942fcae02a344d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:45.125056 7411 tablet_service.cc:1467] Processing CreateTablet for tablet 323a417ca3304694942fcae02a344d50 (DEFAULT_TABLE table=test-table [id=9b881f017db94461bf61b0fe6cb47a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:45.125294 7411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a417ca3304694942fcae02a344d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:45.125397 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.120635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.0.98.2:35867)
0924 07:11:45.120865 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:45.120869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:45.120930 (+ 61us) server_negotiation.cc:407] Connection header received
0924 07:11:45.121060 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:45.121063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:45.121107 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:45.121179 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:45.121497 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.122230 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.123576 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.124441 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.124544 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.124606 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:45.125210 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 07:11:45.125301 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260924 07:11:45.127696 7548 tablet_service.cc:1467] Processing CreateTablet for tablet 323a417ca3304694942fcae02a344d50 (DEFAULT_TABLE table=test-table [id=9b881f017db94461bf61b0fe6cb47a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:45.127905 7854 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Bootstrap starting.
I20260924 07:11:45.128041 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323a417ca3304694942fcae02a344d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:45.128875 7854 tablet_bootstrap.cc:654] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:45.129532 7853 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: Bootstrap starting.
I20260924 07:11:45.130404 7853 tablet_bootstrap.cc:654] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:45.130505 7856 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Bootstrap starting.
I20260924 07:11:45.131065 7854 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: No bootstrap required, opened a new log
I20260924 07:11:45.131145 7854 ts_tablet_manager.cc:1397] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:11:45.131289 7856 tablet_bootstrap.cc:654] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:45.131554 7854 raft_consensus.cc:348] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.131651 7854 raft_consensus.cc:374] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:45.131671 7854 raft_consensus.cc:729] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8147ec5745404904be9e1c1f9ddc2de7, State: Initialized, Role: FOLLOWER
I20260924 07:11:45.132167 7854 consensus_queue.cc:260] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.132331 7856 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: No bootstrap required, opened a new log
I20260924 07:11:45.132418 7856 ts_tablet_manager.cc:1397] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:45.132526 7854 ts_tablet_manager.cc:1428] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:45.132793 7853 tablet_bootstrap.cc:492] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: No bootstrap required, opened a new log
I20260924 07:11:45.132905 7853 ts_tablet_manager.cc:1397] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:45.132854 7856 raft_consensus.cc:348] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.132957 7856 raft_consensus.cc:374] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:45.133018 7856 raft_consensus.cc:729] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ce379435aa4f5e8b550c50cd93009c, State: Initialized, Role: FOLLOWER
I20260924 07:11:45.133169 7856 consensus_queue.cc:260] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.133225 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.132942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:46147)
0924 07:11:45.133101 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.133154 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 07:11:45.133322 7853 raft_consensus.cc:348] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.133448 7853 raft_consensus.cc:374] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:45.133496 7853 raft_consensus.cc:729] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b3a300362fc43e0ae7cda13d60ba0c3, State: Initialized, Role: FOLLOWER
I20260924 07:11:45.133540 7856 ts_tablet_manager.cc:1428] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:45.133664 7853 consensus_queue.cc:260] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.134085 7853 ts_tablet_manager.cc:1428] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:45.135273 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.135035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:60561)
0924 07:11:45.135154 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.135208 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:11:45.136193 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.135824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:37447)
0924 07:11:45.136075 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.136127 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260924 07:11:45.137212 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.136988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:47537)
0924 07:11:45.137100 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.137153 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:11:45.137284 7752 heartbeater.cc:503] Master 127.0.98.61:38141 was elected leader, sending a full tablet report...
I20260924 07:11:45.137706 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.137073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:45479)
0924 07:11:45.137594 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.137643 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459}
I20260924 07:11:45.138267 7616 heartbeater.cc:503] Master 127.0.98.61:38141 was elected leader, sending a full tablet report...
I20260924 07:11:45.138677 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.138485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:35571)
0924 07:11:45.138568 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.138615 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 07:11:45.165866 7753 tablet.cc:2369] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:45.169246 7861 raft_consensus.cc:482] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:45.169389 7861 raft_consensus.cc:504] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.169854 7861 leader_election.cc:288] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263), 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:45.170292 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a417ca3304694942fcae02a344d50" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8147ec5745404904be9e1c1f9ddc2de7" is_pre_election: true
I20260924 07:11:45.170292 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a417ca3304694942fcae02a344d50" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ce379435aa4f5e8b550c50cd93009c" is_pre_election: true
I20260924 07:11:45.170451 7568 raft_consensus.cc:2454] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 0.
I20260924 07:11:45.170451 7704 raft_consensus.cc:2454] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 0.
I20260924 07:11:45.170802 7364 leader_election.cc:302] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b3a300362fc43e0ae7cda13d60ba0c3, 8147ec5745404904be9e1c1f9ddc2de7; no voters:
I20260924 07:11:45.170989 7861 raft_consensus.cc:2781] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:45.171049 7861 raft_consensus.cc:482] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:45.171084 7861 raft_consensus.cc:3037] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:45.172101 7861 raft_consensus.cc:504] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.172436 7861 leader_election.cc:288] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 election: Requested vote from peers 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263), 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867)
I20260924 07:11:45.172803 7568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a417ca3304694942fcae02a344d50" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ce379435aa4f5e8b550c50cd93009c"
I20260924 07:11:45.172940 7568 raft_consensus.cc:3037] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:45.172926 7704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "323a417ca3304694942fcae02a344d50" candidate_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8147ec5745404904be9e1c1f9ddc2de7"
I20260924 07:11:45.173035 7704 raft_consensus.cc:3037] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:45.174053 7704 raft_consensus.cc:2454] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 1.
I20260924 07:11:45.174053 7568 raft_consensus.cc:2454] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b3a300362fc43e0ae7cda13d60ba0c3 in term 1.
I20260924 07:11:45.174360 7364 leader_election.cc:302] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b3a300362fc43e0ae7cda13d60ba0c3, 8147ec5745404904be9e1c1f9ddc2de7; no voters:
I20260924 07:11:45.174530 7861 raft_consensus.cc:2781] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:45.174741 7861 raft_consensus.cc:686] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 LEADER]: Becoming Leader. State: Replica: 5b3a300362fc43e0ae7cda13d60ba0c3, State: Running, Role: LEADER
I20260924 07:11:45.174957 7861 consensus_queue.cc:237] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } }
I20260924 07:11:45.175092 7479 heartbeater.cc:503] Master 127.0.98.61:38141 was elected leader, sending a full tablet report...
I20260924 07:11:45.175544 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.175214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:37759)
0924 07:11:45.175421 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.175477 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 07:11:45.176724 7112 catalog_manager.cc:5495] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b3a300362fc43e0ae7cda13d60ba0c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:45.189930 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.186755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.98.1:37029)
0924 07:11:45.186880 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:45.186884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:45.186918 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:11:45.187042 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:45.187045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:45.187094 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:45.187170 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:45.187347 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.188157 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.189069 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:45.189280 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:45.189356 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:45.189523 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:45.189586 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:45.189631 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:45.189756 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:11:45.189802 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:11:45.191882 7568 raft_consensus.cc:1261] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Refusing update from remote peer 5b3a300362fc43e0ae7cda13d60ba0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:45.192229 7704 raft_consensus.cc:1261] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Refusing update from remote peer 5b3a300362fc43e0ae7cda13d60ba0c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:45.192282 7861 consensus_queue.cc:1035] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ce379435aa4f5e8b550c50cd93009c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:45.192602 7861 consensus_queue.cc:1035] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8147ec5745404904be9e1c1f9ddc2de7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:45.194016 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.193699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:49499)
0924 07:11:45.193841 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.193895 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 07:11:45.194681 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.193424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:56435)
0924 07:11:45.194554 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.194612 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1052}
I20260924 07:11:45.197206 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.196979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:58431)
0924 07:11:45.197084 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.197137 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:11:45.198704 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.198343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.3:39181)
0924 07:11:45.198590 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.198638 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260924 07:11:45.199848 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.199476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.2:52351)
0924 07:11:45.199730 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.199779 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 07:11:45.200430 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.200012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45079 (local address 127.0.98.1:36645)
0924 07:11:45.200319 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.200368 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260924 07:11:45.203080 7112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:45.203212 7112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:45.206640 7112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08d3f7f2168045999a923578819a9207: Sending DeleteTablet for 3 replicas of tablet 323a417ca3304694942fcae02a344d50
I20260924 07:11:45.207358 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet 323a417ca3304694942fcae02a344d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44780
I20260924 07:11:45.207461 7411 tablet_service.cc:1514] Processing DeleteTablet for tablet 323a417ca3304694942fcae02a344d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44334
I20260924 07:11:45.207861 7684 tablet_service.cc:1514] Processing DeleteTablet for tablet 323a417ca3304694942fcae02a344d50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:45 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50326
I20260924 07:11:45.207964 7875 tablet_replica.cc:330] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: stopping tablet replica
I20260924 07:11:45.207965 7876 tablet_replica.cc:330] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3: stopping tablet replica
I20260924 07:11:45.208079 7876 raft_consensus.cc:2229] T 323a417ca3304694942fcae02a344d50 P 5b3a300362fc43e0ae7cda13d60ba0c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:45.208079 7875 raft_consensus.cc:2229] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:45.208112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7258
I20260924 07:11:45.209030 7875 raft_consensus.cc:2258] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:45.209669 7875 ts_tablet_manager.cc:1905] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:45.210088 7877 tablet_replica.cc:330] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: stopping tablet replica
I20260924 07:11:45.210212 7877 raft_consensus.cc:2229] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:45.210335 7877 raft_consensus.cc:2258] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:45.210940 7877 ts_tablet_manager.cc:1905] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:45.211889 7875 ts_tablet_manager.cc:1918] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:45.211958 7875 log.cc:1196] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/323a417ca3304694942fcae02a344d50
I20260924 07:11:45.212173 7875 ts_tablet_manager.cc:1939] T 323a417ca3304694942fcae02a344d50 P 30ce379435aa4f5e8b550c50cd93009c: Deleting consensus metadata
I20260924 07:11:45.213176 7101 catalog_manager.cc:4841] TS 30ce379435aa4f5e8b550c50cd93009c (127.0.98.2:35867): tablet 323a417ca3304694942fcae02a344d50 (table test-table [id=9b881f017db94461bf61b0fe6cb47a35]) successfully deleted
I20260924 07:11:45.213318 7877 ts_tablet_manager.cc:1918] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:45.213425 7877 log.cc:1196] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/323a417ca3304694942fcae02a344d50
I20260924 07:11:45.213670 7877 ts_tablet_manager.cc:1939] T 323a417ca3304694942fcae02a344d50 P 8147ec5745404904be9e1c1f9ddc2de7: Deleting consensus metadata
I20260924 07:11:45.214280 7098 catalog_manager.cc:4841] TS 8147ec5745404904be9e1c1f9ddc2de7 (127.0.98.3:45263): tablet 323a417ca3304694942fcae02a344d50 (table test-table [id=9b881f017db94461bf61b0fe6cb47a35]) successfully deleted
I20260924 07:11:45.216074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7485
I20260924 07:11:45.216306 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:45.216048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37029 (local address 127.0.0.1:44358)
0924 07:11:45.216168 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:45.216221 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:11:45.217453 7099 catalog_manager.cc:4584] TS 5b3a300362fc43e0ae7cda13d60ba0c3 (127.0.98.1:37029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 323a417ca3304694942fcae02a344d50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37029: connect: Connection refused (error 111)
I20260924 07:11:45.221407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7621
I20260924 07:11:45.226195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 6978
I20260924 07:11:45.230991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7080
2026-09-24T07:11:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6810 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:45 dist-test-slave-cnv3 krb5kdc[7880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:45 dist-test-slave-cnv3 krb5kdc[7880](info): set up 2 sockets
Sep 24 07:11:45 dist-test-slave-cnv3 krb5kdc[7880](info): commencing operation
krb5kdc: starting...
W20260924 07:11:46.195669 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 07:11:46.315582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:46.423966 7893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:46.424250 7893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:46.424309 7893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:46.428385 7893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:46.428442 7893 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:11:46.428465 7893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:46.428498 7893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:46.428529 7893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:46.428555 7893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:46.433955 7893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38321
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:46.435075 7893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:46.436153 7893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:46.444985 7899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:46.444989 7901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:46.444985 7898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:46.445029 7893 server_base.cc:1034] running on GCE node
I20260924 07:11:46.445783 7893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:46.446959 7893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:46.448150 7893 hybrid_clock.cc:648] HybridClock initialized: now 1790233906448131 us; error 41 us; skew 500 ppm
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:46.450922 7893 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 07:11:46.451954 7893 webserver.cc:460] Webserver started at http://127.0.98.62:43255/ using document root <none> and password file <none>
I20260924 07:11:46.452519 7893 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:46.452579 7893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:46.452770 7893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:46.454908 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "db5ca48dfa6546e5a8f887b195e498ca"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "5121068010d2293b90a09090a5e88395"
server_key_iv: "b7d9ee9655fb4ec0039fdae30a5271ec"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.455480 7893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "db5ca48dfa6546e5a8f887b195e498ca"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "5121068010d2293b90a09090a5e88395"
server_key_iv: "b7d9ee9655fb4ec0039fdae30a5271ec"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.459249 7893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:11:46.461730 7908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:46.463083 7893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:46.463192 7893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "db5ca48dfa6546e5a8f887b195e498ca"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "5121068010d2293b90a09090a5e88395"
server_key_iv: "b7d9ee9655fb4ec0039fdae30a5271ec"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.463304 7893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:46.476032 7893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:46.479352 7893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:46.479574 7893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:46.487370 7893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38321
I20260924 07:11:46.487406 7970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38321 every 8 connection(s)
I20260924 07:11:46.488381 7893 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:46.491184 7971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:46.491530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7893
I20260924 07:11:46.491641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:46.491899 392 external_mini_cluster.cc:1428] Setting key 7b0b2caa3af80311ba8ababa8fc2a9bf
I20260924 07:11:46.498287 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Bootstrap starting.
I20260924 07:11:46.500905 7971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Neither blocks nor log segments found. Creating new log.
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:46.501756 7971 log.cc:824] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:46.504202 7971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: No bootstrap required, opened a new log
I20260924 07:11:46.507290 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:46.493272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.0.98.62:38321)
0924 07:11:46.493693 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:46.493707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:46.493743 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:11:46.494426 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:46.494444 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:46.494790 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:46.495066 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:46.495985 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.497033 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.498196 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.498764 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.498889 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:46.501627 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:46.501633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:46.501647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:46.501674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:46.504103 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.504620 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.504627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.504633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.504780 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.505015 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.505018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.505020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.505528 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:46.505725 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:46.506008 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 07:11:46.506314 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":284,"thread_start_us":141,"threads_started":1}
I20260924 07:11:46.508172 7971 raft_consensus.cc:348] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } }
I20260924 07:11:46.508459 7971 raft_consensus.cc:374] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:46.508504 7971 raft_consensus.cc:729] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db5ca48dfa6546e5a8f887b195e498ca, State: Initialized, Role: FOLLOWER
I20260924 07:11:46.508931 7971 consensus_queue.cc:260] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } }
I20260924 07:11:46.509050 7971 raft_consensus.cc:388] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:46.509109 7971 raft_consensus.cc:482] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:46.509200 7971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:46.510237 7971 raft_consensus.cc:504] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } }
I20260924 07:11:46.510591 7971 leader_election.cc:302] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db5ca48dfa6546e5a8f887b195e498ca; no voters:
I20260924 07:11:46.510888 7971 leader_election.cc:288] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:46.511008 7976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:46.511221 7976 raft_consensus.cc:686] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [term 1 LEADER]: Becoming Leader. State: Replica: db5ca48dfa6546e5a8f887b195e498ca, State: Running, Role: LEADER
I20260924 07:11:46.511530 7976 consensus_queue.cc:237] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } }
I20260924 07:11:46.512121 7971 sys_catalog.cc:564] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:46.513127 7978 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader db5ca48dfa6546e5a8f887b195e498ca. Latest consensus state: current_term: 1 leader_uuid: "db5ca48dfa6546e5a8f887b195e498ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } } }
I20260924 07:11:46.513255 7978 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:46.513839 7985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:46.514031 7977 sys_catalog.cc:455] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db5ca48dfa6546e5a8f887b195e498ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db5ca48dfa6546e5a8f887b195e498ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38321 } } }
I20260924 07:11:46.514436 7977 sys_catalog.cc:458] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:46.516744 7985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:46.522132 7985 catalog_manager.cc:1347] Generated new cluster ID: 5888d43104474efda66387c75b9c11f9
I20260924 07:11:46.522214 7985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:46.581521 7985 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:46.582553 7985 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:46.590719 7985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Generated new TSK 0
I20260924 07:11:46.591626 7985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 07:11:46.626578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38321
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:46.738802 7997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:46.739090 7997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:46.739135 7997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:46.743263 7997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:46.743335 7997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:46.743395 7997 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:46.743430 7997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:46.748783 7997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:46.749854 7997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:46.750834 7997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:46.760214 8002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:46.760320 7997 server_base.cc:1034] running on GCE node
W20260924 07:11:46.760231 8005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:46.760557 8003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:46.760938 7997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:46.761664 7997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:46.762814 7997 hybrid_clock.cc:648] HybridClock initialized: now 1790233906762743 us; error 84 us; skew 500 ppm
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:46.766176 7997 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:11:46.767112 7997 webserver.cc:460] Webserver started at http://127.0.98.1:39561/ using document root <none> and password file <none>
I20260924 07:11:46.767726 7997 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:46.767789 7997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:46.767953 7997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:46.770036 7997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "87e8b13414cc4976ba49215a3b1efddd"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "34fb198c581ed793cf6184b114c44f91"
server_key_iv: "dc197e4201ec27d406afa42c92343340"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.770550 7997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "87e8b13414cc4976ba49215a3b1efddd"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "34fb198c581ed793cf6184b114c44f91"
server_key_iv: "dc197e4201ec27d406afa42c92343340"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.774204 7997 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 07:11:46.776722 8012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:46.778090 7997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:46.778203 7997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "87e8b13414cc4976ba49215a3b1efddd"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "34fb198c581ed793cf6184b114c44f91"
server_key_iv: "dc197e4201ec27d406afa42c92343340"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.778302 7997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:46.791294 7997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:46.794548 7997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:46.794730 7997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:46.796324 7997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:46.796378 7997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:46.796433 7997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:46.796469 7997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:46.806684 7997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43549
I20260924 07:11:46.806833 8127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43549 every 8 connection(s)
I20260924 07:11:46.807732 7997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:46.809827 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:46.797282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47353 (local address 127.0.98.62:38321)
0924 07:11:46.797402 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:46.797405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:46.798506 (+ 1101us) server_negotiation.cc:407] Connection header received
0924 07:11:46.799645 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:46.799649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:46.799721 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:46.799820 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:46.801057 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.801598 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.802879 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.803212 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.803314 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:46.806284 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:46.806288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:46.806291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:46.806318 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:46.807810 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.808360 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.808364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.808366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.808456 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.808769 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.808772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.808774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.808963 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:46.809046 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:46.809523 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 07:11:46.809661 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:11:46.810613 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:46.797816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.1:47353)
0924 07:11:46.798337 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:46.798384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:46.799362 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:46.800003 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:46.800012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:46.800377 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:46.800905 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:46.800918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.801716 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.801720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:46.802772 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:46.802784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.803438 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.803443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:46.803655 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:46.804331 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:46.804353 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:46.806118 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:46.807917 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:46.807923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:46.807939 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:46.808244 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:46.808546 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:46.808549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:46.808551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:46.808683 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:46.809066 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:46.809072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:46.809304 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 07:11:46.809464 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 07:11:46.809779 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260924 07:11:46.811822 8128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38321
I20260924 07:11:46.811959 8128 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:46.812451 8128 heartbeater.cc:511] Master 127.0.98.62:38321 requested a full tablet report, sending...
I20260924 07:11:46.812745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 7997
I20260924 07:11:46.812860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:46.813179 392 external_mini_cluster.cc:1428] Setting key 1ed133a67234fdb9e54bae9b3eee65bb
I20260924 07:11:46.814090 7925 ts_manager.cc:194] Registered new tserver with Master: 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549)
I20260924 07:11:46.815330 7925 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47353
I20260924 07:11:46.822140 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:46.815540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37137 (local address 127.0.98.62:38321)
0924 07:11:46.815685 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:46.815690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:46.815705 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:46.815740 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:46.815743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:46.815780 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:46.815866 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:46.816381 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.816919 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.817719 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.817930 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.818016 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:46.819249 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:46.819252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:46.819254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:46.819277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:46.820615 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.821022 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.821026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.821028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.821110 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:46.821350 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:46.821353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:46.821355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:46.821532 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:46.821620 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:46.821866 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 07:11:46.821968 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:11:46.822150 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:46.814123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.1:37137)
0924 07:11:46.814399 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:46.814413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:46.814496 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:46.815904 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:46.815907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:46.816122 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:46.816242 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:46.816249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.817048 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.817051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:46.817610 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:46.817617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:46.818031 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:46.818035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:46.818135 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:46.818670 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:46.818683 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:46.819106 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:46.820739 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:46.820743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:46.820745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:46.820935 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:46.821191 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:46.821194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:46.821196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:46.821275 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:46.821623 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:46.821627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:46.821721 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 07:11:46.821805 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:11:46.821962 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 07:11:46.840600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38321
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:46.951604 8137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:46.951942 8137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:46.951996 8137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:46.956367 8137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:46.956451 8137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:46.956511 8137 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:46.956568 8137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:46.962033 8137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:46.963119 8137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:46.964214 8137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:46.974119 8142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:46.974227 8137 server_base.cc:1034] running on GCE node
W20260924 07:11:46.974176 8145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:46.974370 8143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:46.974768 8137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:46.975358 8137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:46.976553 8137 hybrid_clock.cc:648] HybridClock initialized: now 1790233906976533 us; error 32 us; skew 500 ppm
Sep 24 07:11:46 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:46.979234 8137 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:11:46.980196 8137 webserver.cc:460] Webserver started at http://127.0.98.2:33821/ using document root <none> and password file <none>
I20260924 07:11:46.980751 8137 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:46.980819 8137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:46.981005 8137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:46.983284 8137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c26b440205b48eaa720cba94506a70d"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "6295efc448b7701438d8dc0281bed286"
server_key_iv: "e1a629c4bd33f73ce3176191e654989b"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.983872 8137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c26b440205b48eaa720cba94506a70d"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "6295efc448b7701438d8dc0281bed286"
server_key_iv: "e1a629c4bd33f73ce3176191e654989b"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.987605 8137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:11:46.990049 8152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:46.991251 8137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:46.991359 8137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "2c26b440205b48eaa720cba94506a70d"
format_stamp: "Formatted at 2026-09-24 07:11:46 on dist-test-slave-cnv3"
server_key: "6295efc448b7701438d8dc0281bed286"
server_key_iv: "e1a629c4bd33f73ce3176191e654989b"
server_key_version: "encryptionkey@0"
I20260924 07:11:46.991492 8137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:47.009899 8137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:47.013430 8137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:47.013633 8137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:47.015239 8137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:47.015290 8137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:47.015344 8137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:47.015398 8137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:47.025271 8137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43579
I20260924 07:11:47.025331 8267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43579 every 8 connection(s)
I20260924 07:11:47.026289 8137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:47.027137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8137
I20260924 07:11:47.027242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:47.027570 392 external_mini_cluster.cc:1428] Setting key 48bfc5ee629d5a3e12f2f628ab94f8ac
I20260924 07:11:47.029047 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.016088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50259 (local address 127.0.98.62:38321)
0924 07:11:47.016220 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.016225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.017176 (+ 951us) server_negotiation.cc:407] Connection header received
0924 07:11:47.018296 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.018300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.018362 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.018462 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.019962 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.020666 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.021946 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.022165 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.022255 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.025151 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.025155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.025158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.025184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.026601 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.027312 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.027316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.027318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.027446 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.027819 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.027823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.027825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.028067 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.028159 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.028743 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.028893 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:11:47.029906 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.016524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.2:50259)
0924 07:11:47.017000 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.017052 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.018005 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.018712 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.018722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.019155 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:47.019817 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.019833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.020789 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.020793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.021821 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.021831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.022309 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.022313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.022484 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.023176 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:47.023200 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:47.024961 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:47.026712 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.026718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.026735 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.027198 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.027548 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.027551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.027553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.027718 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.028164 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:47.028170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:47.028465 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.028674 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.029026 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":98,"threads_started":1}
I20260924 07:11:47.031170 8268 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38321
I20260924 07:11:47.031332 8268 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:47.031872 8268 heartbeater.cc:511] Master 127.0.98.62:38321 requested a full tablet report, sending...
I20260924 07:11:47.032968 7923 ts_manager.cc:194] Registered new tserver with Master: 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579)
I20260924 07:11:47.033716 7923 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50259
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 07:11:47.042323 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.033625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.2:47061)
0924 07:11:47.033953 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.033968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.034066 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.035481 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.035484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.035811 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:47.035928 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.035936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.037019 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.037022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.037676 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.037689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.038121 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.038125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.038214 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.038758 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:47.038770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:47.039097 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:47.040762 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.040773 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.040775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.041014 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.041297 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.041299 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.041301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.041386 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.041775 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:47.041779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:47.041871 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.041957 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.042109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260924 07:11:47.042331 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.033694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47061 (local address 127.0.98.62:38321)
0924 07:11:47.033814 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.033819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.034044 (+ 225us) server_negotiation.cc:407] Connection header received
0924 07:11:47.034176 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.034179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.034228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.034310 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.036066 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.036876 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.037785 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.038011 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.038099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.039233 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.039236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.039238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.039263 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.040651 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.041146 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.041148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.041150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.041221 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.041461 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.041463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.041464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.041678 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.041767 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.042022 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.042200 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:11:47.058363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38321
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:47.168946 8277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:47.169226 8277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:47.169286 8277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:47.173297 8277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:47.173355 8277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:47.173408 8277 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:11:47.173463 8277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:47.178628 8277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:47.179690 8277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:47.180753 8277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:47.190346 8282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:47.190366 8283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:47.190341 8285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:47.190610 8277 server_base.cc:1034] running on GCE node
I20260924 07:11:47.191087 8277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:47.191754 8277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:47.192919 8277 hybrid_clock.cc:648] HybridClock initialized: now 1790233907192872 us; error 59 us; skew 500 ppm
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:47.195957 8277 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:11:47.196877 8277 webserver.cc:460] Webserver started at http://127.0.98.3:42873/ using document root <none> and password file <none>
I20260924 07:11:47.197409 8277 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:47.197466 8277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:47.197633 8277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:47.199725 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d89a41682c544d1a5d1bc58d6825287"
format_stamp: "Formatted at 2026-09-24 07:11:47 on dist-test-slave-cnv3"
server_key: "3586dea83ba87b7c4b3920c77d6afd23"
server_key_iv: "fe6d8d5d69634e67179066b5ecdd3d82"
server_key_version: "encryptionkey@0"
I20260924 07:11:47.200263 8277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d89a41682c544d1a5d1bc58d6825287"
format_stamp: "Formatted at 2026-09-24 07:11:47 on dist-test-slave-cnv3"
server_key: "3586dea83ba87b7c4b3920c77d6afd23"
server_key_iv: "fe6d8d5d69634e67179066b5ecdd3d82"
server_key_version: "encryptionkey@0"
I20260924 07:11:47.204200 8277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:11:47.206849 8292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:47.208217 8277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:47.208325 8277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "2d89a41682c544d1a5d1bc58d6825287"
format_stamp: "Formatted at 2026-09-24 07:11:47 on dist-test-slave-cnv3"
server_key: "3586dea83ba87b7c4b3920c77d6afd23"
server_key_iv: "fe6d8d5d69634e67179066b5ecdd3d82"
server_key_version: "encryptionkey@0"
I20260924 07:11:47.208424 8277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:47.224576 8277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:47.227874 8277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:47.228053 8277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:47.229490 8277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:47.229542 8277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:47.229594 8277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:47.229622 8277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:47.239214 8277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35507
I20260924 07:11:47.239274 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35507 every 8 connection(s)
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233907, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:47.240305 8277 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:47.243875 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.230351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35653 (local address 127.0.98.62:38321)
0924 07:11:47.230487 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.230492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.231356 (+ 864us) server_negotiation.cc:407] Connection header received
0924 07:11:47.232484 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.232487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.232540 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.232619 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.234240 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.235060 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.236772 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.236985 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.237067 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.240190 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.240194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.240196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.240223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.241748 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.242305 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.242309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.242311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.242391 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.242760 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.242763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.242765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.242956 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.243037 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.243585 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.243708 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:47.244565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8277
I20260924 07:11:47.244707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:47.244827 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.230737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.3:35653)
0924 07:11:47.231171 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.231222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.232263 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.232838 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.232848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.233310 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:47.234034 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.234055 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.235189 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.235193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.236630 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.236640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.237140 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.237145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.237389 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.238099 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:47.238122 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:47.239987 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:47.241866 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.241873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.241888 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.242204 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.242471 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.242474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.242476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.242659 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.243049 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:47.243056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:47.243309 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.243517 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.243846 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260924 07:11:47.245012 392 external_mini_cluster.cc:1428] Setting key 1facf4821182515661130aed5740d709
I20260924 07:11:47.246243 8408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38321
I20260924 07:11:47.246393 8408 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:47.246968 8408 heartbeater.cc:511] Master 127.0.98.62:38321 requested a full tablet report, sending...
I20260924 07:11:47.248030 7924 ts_manager.cc:194] Registered new tserver with Master: 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.248934 7924 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35653
I20260924 07:11:47.255622 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.248219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.98.3:38331)
0924 07:11:47.248495 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.248508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.248595 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.248847 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.248849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.249024 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:47.249182 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.249188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.250157 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.250161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.250774 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.250783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.251368 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.251371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.251500 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.252065 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:47.252077 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:47.252486 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:47.254046 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.254049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.254051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.254298 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.254566 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.254568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.254569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.254647 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.255062 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:47.255064 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:47.255148 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.255281 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.255459 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260924 07:11:47.255661 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.248291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38331 (local address 127.0.98.62:38321)
0924 07:11:47.248452 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.248457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.248572 (+ 115us) server_negotiation.cc:407] Connection header received
0924 07:11:47.248660 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.248663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.248707 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.248800 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.249302 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.249989 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.250913 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.251235 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.251351 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.252643 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.252646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.252648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.252672 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.253941 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.254401 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.254404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.254405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.254485 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.254765 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.254770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.254771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.254982 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.255081 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.255411 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.255516 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:11:47.258671 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:11:47.282238 7974 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:47.275681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.0.98.62:38321)
0924 07:11:47.275812 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.275815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.275829 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:11:47.275870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.275873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.275911 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.275990 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.276702 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.277319 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.278227 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.278462 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.278548 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.282076 (+ 3528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37980: BlockingRecv error: recv got EOF from 127.0.0.1:37980 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233907, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:47.291661 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.283786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.62:38321)
0924 07:11:47.283933 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.283938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.284038 (+ 100us) server_negotiation.cc:407] Connection header received
0924 07:11:47.284219 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.284222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.284270 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.284338 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.284951 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.285470 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.286247 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.286464 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.286549 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.288904 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.288908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.288911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.288940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.290110 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.290490 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.290494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.290496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.290571 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.290802 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.290805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.290807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.290993 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.291107 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.291335 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.291477 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:11:47.294821 7924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:47.297863 7924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:47.312492 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.306005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43579 (local address 127.0.0.1:38402)
0924 07:11:47.307002 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.307013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.307150 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.307966 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.307970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.307988 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.308140 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.308145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.309472 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.309475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.310711 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.310720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.311893 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.311896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.312030 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.312086 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.312183 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.312318 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":45,"spinlock_wait_cycles":6400,"thread_start_us":176,"threads_started":1}
I20260924 07:11:47.312950 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.306254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.0.98.2:43579)
0924 07:11:47.306629 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.306642 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.307291 (+ 649us) server_negotiation.cc:407] Connection header received
0924 07:11:47.307369 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.307413 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.307646 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.307805 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.308251 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.309366 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.310845 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.311787 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.311885 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.312114 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.312704 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.312811 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":104,"threads_started":1}
I20260924 07:11:47.313726 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.306093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.0.98.1:43549)
0924 07:11:47.306640 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.306658 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.307291 (+ 633us) server_negotiation.cc:407] Connection header received
0924 07:11:47.307370 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.307411 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.307644 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.307805 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.308107 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.309339 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.312144 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.312992 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.313093 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.313233 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.313507 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.313602 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":79,"threads_started":1}
I20260924 07:11:47.313784 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.305972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43549 (local address 127.0.0.1:47742)
0924 07:11:47.306949 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.307002 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.307150 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.307804 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.307810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.307832 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.307970 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.307978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.310840 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.310843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.312010 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.312018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.313290 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.313294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.313382 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.313405 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.313512 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.313592 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":3,"spinlock_wait_cycles":2688,"thread_start_us":92,"threads_started":1}
I20260924 07:11:47.313805 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.306740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35507 (local address 127.0.0.1:60662)
0924 07:11:47.307249 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.307262 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.307336 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.307999 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.308001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.308014 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.308138 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.308145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.310477 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.310480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.311689 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.311699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.313328 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.313332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.313421 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.313443 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.313549 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.313638 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":33,"thread_start_us":78,"threads_started":1}
I20260924 07:11:47.314946 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.306894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.0.98.3:35507)
0924 07:11:47.307592 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.307600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.307618 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:11:47.307692 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.307695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.307872 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.308020 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.309493 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.310374 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.311824 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.313013 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.313113 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.313269 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.314617 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.314787 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":100,"threads_started":1}
I20260924 07:11:47.316187 8186 tablet_service.cc:1467] Processing CreateTablet for tablet a4f12307fbfb4495a59be4f95df3f11c (DEFAULT_TABLE table=test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.316293 8062 tablet_service.cc:1467] Processing CreateTablet for tablet a4f12307fbfb4495a59be4f95df3f11c (DEFAULT_TABLE table=test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.317203 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f12307fbfb4495a59be4f95df3f11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.317204 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f12307fbfb4495a59be4f95df3f11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.323946 8437 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Bootstrap starting.
I20260924 07:11:47.325358 8342 tablet_service.cc:1467] Processing CreateTablet for tablet a4f12307fbfb4495a59be4f95df3f11c (DEFAULT_TABLE table=test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.326241 8437 tablet_bootstrap.cc:654] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.326488 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4f12307fbfb4495a59be4f95df3f11c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.326478 8436 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Bootstrap starting.
I20260924 07:11:47.327154 8437 log.cc:824] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:47.328625 8436 tablet_bootstrap.cc:654] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.329779 8436 log.cc:824] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:47.330130 8437 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: No bootstrap required, opened a new log
I20260924 07:11:47.330353 8437 ts_tablet_manager.cc:1397] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:11:47.331653 8436 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: No bootstrap required, opened a new log
I20260924 07:11:47.331851 8436 ts_tablet_manager.cc:1397] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:11:47.334319 8440 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Bootstrap starting.
I20260924 07:11:47.334398 8437 raft_consensus.cc:348] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.334695 8437 raft_consensus.cc:374] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.334754 8437 raft_consensus.cc:729] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.334870 8436 raft_consensus.cc:348] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.335218 8436 raft_consensus.cc:374] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.335276 8436 raft_consensus.cc:729] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.335212 8437 consensus_queue.cc:260] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.335750 8436 consensus_queue.cc:260] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.335883 8268 heartbeater.cc:503] Master 127.0.98.62:38321 was elected leader, sending a full tablet report...
I20260924 07:11:47.336824 8436 ts_tablet_manager.cc:1428] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:11:47.336825 8437 ts_tablet_manager.cc:1428] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:11:47.337170 8128 heartbeater.cc:503] Master 127.0.98.62:38321 was elected leader, sending a full tablet report...
I20260924 07:11:47.337460 8440 tablet_bootstrap.cc:654] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.338189 8440 log.cc:824] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:47.340703 8440 tablet_bootstrap.cc:492] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: No bootstrap required, opened a new log
I20260924 07:11:47.340893 8440 ts_tablet_manager.cc:1397] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 07:11:47.343777 8440 raft_consensus.cc:348] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.344038 8440 raft_consensus.cc:374] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.344094 8440 raft_consensus.cc:729] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89a41682c544d1a5d1bc58d6825287, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.344537 8440 consensus_queue.cc:260] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.345261 8408 heartbeater.cc:503] Master 127.0.98.62:38321 was elected leader, sending a full tablet report...
I20260924 07:11:47.345492 8440 ts_tablet_manager.cc:1428] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:11:47.348951 8442 raft_consensus.cc:482] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:47.349121 8442 raft_consensus.cc:504] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.350229 8442 leader_election.cc:288] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.354223 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.350551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45893 (local address 127.0.98.3:35507)
0924 07:11:47.350670 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.350674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.350992 (+ 318us) server_negotiation.cc:407] Connection header received
0924 07:11:47.351092 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.351094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.351134 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.351210 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.351533 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.352083 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353309 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.353699 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353789 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.353853 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.354030 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.354098 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:11:47.354240 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.350661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41659 (local address 127.0.98.2:43579)
0924 07:11:47.350789 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.350792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.351245 (+ 453us) server_negotiation.cc:407] Connection header received
0924 07:11:47.351339 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.351342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.351421 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.351486 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.351717 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.352243 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353319 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.353724 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353813 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.353870 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.354047 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.354117 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:47.354269 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.350455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35507 (local address 127.0.98.1:45893)
0924 07:11:47.350911 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.350923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.351019 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.351244 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.351247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.351266 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.351411 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.351418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.352191 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.352195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.353196 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.353206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.353803 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.353900 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.353942 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.354013 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.354089 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":62,"thread_start_us":129,"threads_started":1}
I20260924 07:11:47.354276 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.350455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43579 (local address 127.0.98.1:41659)
0924 07:11:47.351162 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.351171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.351246 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.351496 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.351498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.351513 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.351615 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.351621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.352350 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.352353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.353210 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.353219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.353827 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.353831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.353946 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.353962 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.354023 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.354096 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":68,"thread_start_us":114,"threads_started":1}
I20260924 07:11:47.355144 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f12307fbfb4495a59be4f95df3f11c" candidate_uuid: "87e8b13414cc4976ba49215a3b1efddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287" is_pre_election: true
I20260924 07:11:47.355224 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f12307fbfb4495a59be4f95df3f11c" candidate_uuid: "87e8b13414cc4976ba49215a3b1efddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c26b440205b48eaa720cba94506a70d" is_pre_election: true
I20260924 07:11:47.355499 8362 raft_consensus.cc:2454] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e8b13414cc4976ba49215a3b1efddd in term 0.
I20260924 07:11:47.355520 8222 raft_consensus.cc:2454] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e8b13414cc4976ba49215a3b1efddd in term 0.
I20260924 07:11:47.355945 8016 leader_election.cc:302] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 87e8b13414cc4976ba49215a3b1efddd; no voters:
I20260924 07:11:47.356179 8442 raft_consensus.cc:2781] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:47.356252 8442 raft_consensus.cc:482] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:47.356302 8442 raft_consensus.cc:3037] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.357267 8442 raft_consensus.cc:504] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.357621 8442 leader_election.cc:288] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [CANDIDATE]: Term 1 election: Requested vote from peers 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.357926 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f12307fbfb4495a59be4f95df3f11c" candidate_uuid: "87e8b13414cc4976ba49215a3b1efddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287"
I20260924 07:11:47.357936 8222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4f12307fbfb4495a59be4f95df3f11c" candidate_uuid: "87e8b13414cc4976ba49215a3b1efddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c26b440205b48eaa720cba94506a70d"
I20260924 07:11:47.358049 8362 raft_consensus.cc:3037] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.358049 8222 raft_consensus.cc:3037] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.359308 8362 raft_consensus.cc:2454] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e8b13414cc4976ba49215a3b1efddd in term 1.
I20260924 07:11:47.359637 8013 leader_election.cc:302] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d89a41682c544d1a5d1bc58d6825287, 87e8b13414cc4976ba49215a3b1efddd; no voters:
I20260924 07:11:47.359665 8222 raft_consensus.cc:2454] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87e8b13414cc4976ba49215a3b1efddd in term 1.
I20260924 07:11:47.359846 8442 raft_consensus.cc:2781] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:47.360033 8442 raft_consensus.cc:686] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 1 LEADER]: Becoming Leader. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Running, Role: LEADER
I20260924 07:11:47.360381 8442 consensus_queue.cc:237] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.363487 7924 catalog_manager.cc:5495] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "87e8b13414cc4976ba49215a3b1efddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:47.376541 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.373110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.0.98.1:43549)
0924 07:11:47.373272 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.373277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.373290 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:47.373372 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.373375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.373436 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.373534 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:47.373756 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.374322 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.375156 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.375406 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.375494 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.375737 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:47.376187 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:47.376288 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.376351 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.376409 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:11:47.382478 8362 raft_consensus.cc:1261] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Refusing update from remote peer 87e8b13414cc4976ba49215a3b1efddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:47.382478 8222 raft_consensus.cc:1261] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Refusing update from remote peer 87e8b13414cc4976ba49215a3b1efddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:47.383246 8442 consensus_queue.cc:1035] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:47.383482 8447 consensus_queue.cc:1035] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:47.392148 8451 mvcc.cc:205] Tried to move back new op lower bound from 7332798084630953984 to 7332798084550680576. Current Snapshot: MvccSnapshot[applied={T|T < 7332798084630953984}]
I20260924 07:11:47.394666 8452 mvcc.cc:205] Tried to move back new op lower bound from 7332798084630953984 to 7332798084550680576. Current Snapshot: MvccSnapshot[applied={T|T < 7332798084630953984 or (T in {7332798084630953984})}]
I20260924 07:11:47.400092 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:47.400247 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:47.402590 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Sending DeleteTablet for 3 replicas of tablet a4f12307fbfb4495a59be4f95df3f11c
I20260924 07:11:47.403395 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f12307fbfb4495a59be4f95df3f11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38402
I20260924 07:11:47.403429 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f12307fbfb4495a59be4f95df3f11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60662
I20260924 07:11:47.404358 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:47.404812 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:47.407642 8461 tablet_replica.cc:330] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: stopping tablet replica
I20260924 07:11:47.408118 8461 raft_consensus.cc:2229] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:47.408069 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet a4f12307fbfb4495a59be4f95df3f11c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260924 07:11:47.408617 8461 raft_consensus.cc:2258] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:47.408713 8460 tablet_replica.cc:330] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: stopping tablet replica
I20260924 07:11:47.409454 8460 raft_consensus.cc:2229] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:47.409953 8460 raft_consensus.cc:2258] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:47.411285 8461 ts_tablet_manager.cc:1905] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:47.411862 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 (DEFAULT_TABLE table=test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.411864 8062 tablet_service.cc:1467] Processing CreateTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 (DEFAULT_TABLE table=test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.412145 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062c542b638c41cab8e1591af6c5b2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.412150 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062c542b638c41cab8e1591af6c5b2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.414616 8436 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Bootstrap starting.
I20260924 07:11:47.414711 8437 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Bootstrap starting.
I20260924 07:11:47.414798 8461 ts_tablet_manager.cc:1918] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:47.414938 8461 log.cc:1196] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/a4f12307fbfb4495a59be4f95df3f11c
I20260924 07:11:47.414938 8460 ts_tablet_manager.cc:1905] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:47.415364 8461 ts_tablet_manager.cc:1939] T a4f12307fbfb4495a59be4f95df3f11c P 2c26b440205b48eaa720cba94506a70d: Deleting consensus metadata
I20260924 07:11:47.415686 8437 tablet_bootstrap.cc:654] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.415689 8436 tablet_bootstrap.cc:654] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.416903 7912 catalog_manager.cc:4841] TS 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579): tablet a4f12307fbfb4495a59be4f95df3f11c (table test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]) successfully deleted
I20260924 07:11:47.417053 8342 tablet_service.cc:1467] Processing CreateTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 (DEFAULT_TABLE table=test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:47.417305 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 062c542b638c41cab8e1591af6c5b2a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.417433 8462 tablet_replica.cc:330] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: stopping tablet replica
I20260924 07:11:47.417757 8462 raft_consensus.cc:2229] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:47.418156 8462 raft_consensus.cc:2258] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:47.419523 8440 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Bootstrap starting.
I20260924 07:11:47.419852 8462 ts_tablet_manager.cc:1905] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:47.420511 8440 tablet_bootstrap.cc:654] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.420526 8460 ts_tablet_manager.cc:1918] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:47.420612 8460 log.cc:1196] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/a4f12307fbfb4495a59be4f95df3f11c
I20260924 07:11:47.420893 8460 ts_tablet_manager.cc:1939] T a4f12307fbfb4495a59be4f95df3f11c P 2d89a41682c544d1a5d1bc58d6825287: Deleting consensus metadata
I20260924 07:11:47.422101 7909 catalog_manager.cc:4841] TS 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507): tablet a4f12307fbfb4495a59be4f95df3f11c (table test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]) successfully deleted
I20260924 07:11:47.422637 8462 ts_tablet_manager.cc:1918] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:47.422724 8462 log.cc:1196] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/a4f12307fbfb4495a59be4f95df3f11c
I20260924 07:11:47.422837 8437 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: No bootstrap required, opened a new log
I20260924 07:11:47.422919 8437 ts_tablet_manager.cc:1397] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 07:11:47.422989 8462 ts_tablet_manager.cc:1939] T a4f12307fbfb4495a59be4f95df3f11c P 87e8b13414cc4976ba49215a3b1efddd: Deleting consensus metadata
I20260924 07:11:47.423089 8436 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: No bootstrap required, opened a new log
I20260924 07:11:47.423157 8436 ts_tablet_manager.cc:1397] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 07:11:47.423367 8437 raft_consensus.cc:348] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.423519 8437 raft_consensus.cc:374] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.423548 8437 raft_consensus.cc:729] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.423621 8436 raft_consensus.cc:348] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.423724 8436 raft_consensus.cc:374] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.423696 8437 consensus_queue.cc:260] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.423755 8436 raft_consensus.cc:729] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.423863 8436 consensus_queue.cc:260] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.423941 8437 ts_tablet_manager.cc:1428] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:47.424114 8436 ts_tablet_manager.cc:1428] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:47.424166 7909 catalog_manager.cc:4841] TS 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549): tablet a4f12307fbfb4495a59be4f95df3f11c (table test-table [id=dcfd8fdf599c4fc8ae555b2557f55d4d]) successfully deleted
I20260924 07:11:47.428272 8440 tablet_bootstrap.cc:492] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: No bootstrap required, opened a new log
I20260924 07:11:47.428359 8440 ts_tablet_manager.cc:1397] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260924 07:11:47.428797 8440 raft_consensus.cc:348] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.428896 8440 raft_consensus.cc:374] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.428917 8440 raft_consensus.cc:729] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89a41682c544d1a5d1bc58d6825287, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.429044 8440 consensus_queue.cc:260] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.429282 8440 ts_tablet_manager.cc:1428] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260924 07:11:47.491644 8409 tablet.cc:2369] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:47.526611 8441 raft_consensus.cc:482] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:47.526801 8441 raft_consensus.cc:504] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
W20260924 07:11:47.528134 8269 tablet.cc:2369] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:47.528249 8441 leader_election.cc:288] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.532745 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.528653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43367 (local address 127.0.98.1:43549)
0924 07:11:47.528783 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.528788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.528807 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:11:47.529026 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.529029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.529081 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.529158 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.529503 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.530003 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.531611 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.532249 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.532360 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.532427 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.532563 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.532625 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:11:47.532765 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.528479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43549 (local address 127.0.98.2:43367)
0924 07:11:47.528740 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.528756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.528849 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.529211 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.529215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.529240 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.529363 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.529370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.530135 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.530139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.531463 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.531473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.532349 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.532353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.532452 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.532474 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.532541 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.532613 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260924 07:11:47.533342 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062c542b638c41cab8e1591af6c5b2a2" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd" is_pre_election: true
I20260924 07:11:47.533533 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.528765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35507 (local address 127.0.98.2:49409)
0924 07:11:47.529030 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.529121 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.529263 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.530123 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.530125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.530142 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:47.530250 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.530254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.530931 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.530934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.532050 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.532060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.533106 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.533109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.533200 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.533222 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.533292 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.533355 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260924 07:11:47.533530 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.529012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49409 (local address 127.0.98.3:35507)
0924 07:11:47.529133 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.529138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.529198 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:11:47.529365 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.529367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.529406 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.529468 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:47.530360 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.530832 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.532216 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.532820 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.532928 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.532987 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.533329 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.533407 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:47.533628 8082 raft_consensus.cc:2454] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:47.534044 8153 leader_election.cc:302] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 87e8b13414cc4976ba49215a3b1efddd; no voters:
I20260924 07:11:47.534081 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062c542b638c41cab8e1591af6c5b2a2" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287" is_pre_election: true
I20260924 07:11:47.534202 8362 raft_consensus.cc:2454] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:47.534341 8441 raft_consensus.cc:2781] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:47.534403 8441 raft_consensus.cc:482] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:47.534447 8441 raft_consensus.cc:3037] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.535840 8441 raft_consensus.cc:504] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.536214 8441 leader_election.cc:288] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Requested vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.536628 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062c542b638c41cab8e1591af6c5b2a2" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287"
I20260924 07:11:47.536754 8362 raft_consensus.cc:3037] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.537781 8362 raft_consensus.cc:2454] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:47.538018 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "062c542b638c41cab8e1591af6c5b2a2" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd"
I20260924 07:11:47.538057 8153 leader_election.cc:302] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 2d89a41682c544d1a5d1bc58d6825287; no voters:
I20260924 07:11:47.538136 8082 raft_consensus.cc:3037] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.538225 8441 raft_consensus.cc:2781] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:47.538398 8441 raft_consensus.cc:686] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Becoming Leader. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Running, Role: LEADER
I20260924 07:11:47.538808 8441 consensus_queue.cc:237] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.539237 8082 raft_consensus.cc:2454] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:47.541409 7925 catalog_manager.cc:5495] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c26b440205b48eaa720cba94506a70d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } health_report { overall_health: UNKNOWN } } }
W20260924 07:11:47.559069 8129 tablet.cc:2369] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:11:47 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:47.577289 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.569092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38321 (local address 127.0.0.1:38004)
0924 07:11:47.569435 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:47.569449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:47.569555 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:47.569791 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:47.569793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:47.570179 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:47.570270 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.570276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.571038 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.571041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.571653 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:47.571661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.572082 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.572084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:47.572175 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.572695 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:47.572707 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:47.574204 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:47.575779 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.575787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.575792 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.576010 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.576290 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:47.576292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:47.576293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:47.576364 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:47.576788 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:47.576793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:47.576917 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 07:11:47.577012 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:11:47.577129 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":22,"thread_start_us":166,"threads_started":1}
I20260924 07:11:47.577382 7974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:47.569162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.0.98.62:38321)
0924 07:11:47.569303 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:47.569307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:47.569524 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:11:47.569628 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:47.569631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:47.569675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:47.569755 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:47.570392 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.570940 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.571790 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:47.571995 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:47.572071 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:47.574338 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:47.574341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:47.574343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:47.574364 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:47.575672 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.576085 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.576088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.576089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.576202 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:47.576468 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:47.576471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:47.576473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:47.576661 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:47.576783 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:47.577094 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:11:47.577234 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":61}
I20260924 07:11:47.581482 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:11:47.582262 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:47.587761 8062 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcfd3de4fa7454a90616aedc57d9ec6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67055da91bb046a6a6c0bf2f148cf37a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:47.587908 8342 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcfd3de4fa7454a90616aedc57d9ec6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67055da91bb046a6a6c0bf2f148cf37a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:47.587981 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 6fcfd3de4fa7454a90616aedc57d9ec6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67055da91bb046a6a6c0bf2f148cf37a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:11:47.588198 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcfd3de4fa7454a90616aedc57d9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.588198 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcfd3de4fa7454a90616aedc57d9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.588358 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fcfd3de4fa7454a90616aedc57d9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:47.590672 8437 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: Bootstrap starting.
I20260924 07:11:47.591609 8437 tablet_bootstrap.cc:654] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.592126 8436 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: Bootstrap starting.
I20260924 07:11:47.592713 8437 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: No bootstrap required, opened a new log
I20260924 07:11:47.592801 8437 ts_tablet_manager.cc:1397] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:47.592860 8436 tablet_bootstrap.cc:654] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.593233 8437 raft_consensus.cc:348] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.593345 8437 raft_consensus.cc:374] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.593375 8437 raft_consensus.cc:729] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.593382 8440 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: Bootstrap starting.
I20260924 07:11:47.593490 8437 consensus_queue.cc:260] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.593742 8441 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:47.593886 8441 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:47.593748 8437 ts_tablet_manager.cc:1428] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:47.594293 8440 tablet_bootstrap.cc:654] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:47.595569 8436 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: No bootstrap required, opened a new log
I20260924 07:11:47.595664 8436 ts_tablet_manager.cc:1397] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 07:11:47.595875 8440 tablet_bootstrap.cc:492] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: No bootstrap required, opened a new log
I20260924 07:11:47.595975 8440 ts_tablet_manager.cc:1397] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 07:11:47.596066 8436 raft_consensus.cc:348] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.596187 8436 raft_consensus.cc:374] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.596225 8436 raft_consensus.cc:729] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.596351 8436 consensus_queue.cc:260] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.596408 8440 raft_consensus.cc:348] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.596544 8440 raft_consensus.cc:374] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:47.596589 8440 raft_consensus.cc:729] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89a41682c544d1a5d1bc58d6825287, State: Initialized, Role: FOLLOWER
I20260924 07:11:47.596602 8436 ts_tablet_manager.cc:1428] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:47.596724 8440 consensus_queue.cc:260] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.596789 8442 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:47.596920 8442 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:47.596973 8440 ts_tablet_manager.cc:1428] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:47.596961 8444 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:47.597110 8444 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:47.945480 8441 raft_consensus.cc:482] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:47.945669 8441 raft_consensus.cc:504] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.946081 8441 leader_election.cc:288] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.946606 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcfd3de4fa7454a90616aedc57d9ec6" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd" is_pre_election: true
I20260924 07:11:47.946625 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcfd3de4fa7454a90616aedc57d9ec6" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287" is_pre_election: true
I20260924 07:11:47.946796 8082 raft_consensus.cc:2454] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:47.946846 8362 raft_consensus.cc:2454] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:47.947129 8153 leader_election.cc:302] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 2d89a41682c544d1a5d1bc58d6825287; no voters:
I20260924 07:11:47.947405 8441 raft_consensus.cc:2781] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:47.947477 8441 raft_consensus.cc:482] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:47.947523 8441 raft_consensus.cc:3037] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.948522 8441 raft_consensus.cc:504] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.948845 8441 leader_election.cc:288] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Requested vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:47.949260 8082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcfd3de4fa7454a90616aedc57d9ec6" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd"
I20260924 07:11:47.949303 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fcfd3de4fa7454a90616aedc57d9ec6" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287"
I20260924 07:11:47.949391 8082 raft_consensus.cc:3037] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.949400 8362 raft_consensus.cc:3037] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:47.950481 8362 raft_consensus.cc:2454] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:47.950554 8082 raft_consensus.cc:2454] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:47.950775 8153 leader_election.cc:302] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 2d89a41682c544d1a5d1bc58d6825287; no voters:
I20260924 07:11:47.950941 8441 raft_consensus.cc:2781] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:47.951041 8441 raft_consensus.cc:686] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Becoming Leader. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Running, Role: LEADER
I20260924 07:11:47.951263 8441 consensus_queue.cc:237] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:47.952013 8467 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: TxnStatusTablet state changed. Reason: New leader 2c26b440205b48eaa720cba94506a70d. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:47.952140 8467 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:47.952355 8479 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:11:47.952543 8479 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:11:47.953044 7924 catalog_manager.cc:5495] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c26b440205b48eaa720cba94506a70d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:48.046744 8467 consensus_queue.cc:1035] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.049698 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.045065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.0.98.2:43579)
0924 07:11:48.045315 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.045319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.045334 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:48.045403 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.045406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.045467 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.045564 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:48.045848 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.046593 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.047927 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.048275 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.048382 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.048699 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.049243 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.049337 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:48.049480 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:11:48.049537 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260924 07:11:48.049723 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.044970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43579 (local address 127.0.0.1:38416)
0924 07:11:48.045122 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:48.045139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:48.045251 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:48.045588 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:48.045591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:48.045600 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:48.045703 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:48.045709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.046707 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.046711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:48.047714 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:48.047727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.048377 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.048380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:48.048471 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.048524 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.049378 (+ 854us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.049392 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:11:48.049453 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:11:48.049541 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:11:48.052990 8467 consensus_queue.cc:1035] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.053516 8082 raft_consensus.cc:1261] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.053571 8362 raft_consensus.cc:1261] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.054068 8441 consensus_queue.cc:1035] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.054625 8441 consensus_queue.cc:1035] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.056185 8488 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: TxnStatusTablet state changed. Reason: New leader 2c26b440205b48eaa720cba94506a70d. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.056301 8488 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:48.057629 8444 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: TxnStatusTablet state changed. Reason: New leader 2c26b440205b48eaa720cba94506a70d. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.057757 8444 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:48.058537 8467 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.058656 8467 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:48.059763 8441 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.059873 8441 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2c26b440205b48eaa720cba94506a70d: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:11:48.060177 8444 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.060299 8444 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 2d89a41682c544d1a5d1bc58d6825287: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:48.060803 8488 tablet_replica.cc:439] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } } }
I20260924 07:11:48.060915 8488 tablet_replica.cc:442] T 6fcfd3de4fa7454a90616aedc57d9ec6 P 87e8b13414cc4976ba49215a3b1efddd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:11:48.061668 8487 mvcc.cc:205] Tried to move back new op lower bound from 7332798087380975616 to 7332798086969749504. Current Snapshot: MvccSnapshot[applied={T|T < 7332798087380975616}]
I20260924 07:11:48.068391 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.065157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.0.98.2:43579)
0924 07:11:48.065369 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.065373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.065383 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:11:48.065417 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.065419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.065457 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.065520 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:48.065769 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.066397 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.067298 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.067595 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.067684 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.067968 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.068043 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.068096 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:48.068195 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:11:48.068255 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 07:11:48.080400 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.076534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38307 (local address 127.0.98.2:43579)
0924 07:11:48.076680 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.076684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.076839 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:11:48.076981 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.076983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.077022 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.077082 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:48.077377 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.077897 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.079153 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.079425 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.079511 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.079866 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.079941 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.080001 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:48.080188 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 07:11:48.080266 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:11:48.080435 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.076441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43579 (local address 127.0.98.2:38307)
0924 07:11:48.076765 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:48.076780 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:48.076898 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:48.077128 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:48.077131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:48.077140 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:11:48.077283 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:48.077290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.077998 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.078002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:48.079012 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:48.079023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.079538 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.079542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:48.079644 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.079703 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.080037 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.080044 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:11:48.080119 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:11:48.080215 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
I20260924 07:11:48.130722 7925 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.130898 7925 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.133334 7925 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Sending DeleteTablet for 3 replicas of tablet 062c542b638c41cab8e1591af6c5b2a2
I20260924 07:11:48.134027 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260924 07:11:48.134161 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60662
I20260924 07:11:48.134315 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 062c542b638c41cab8e1591af6c5b2a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38402
I20260924 07:11:48.134645 8504 tablet_replica.cc:330] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: stopping tablet replica
I20260924 07:11:48.134783 8504 raft_consensus.cc:2229] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.134902 8504 raft_consensus.cc:2258] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.135664 8504 ts_tablet_manager.cc:1905] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.138792 8504 ts_tablet_manager.cc:1918] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:48.138875 8504 log.cc:1196] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/062c542b638c41cab8e1591af6c5b2a2
I20260924 07:11:48.139122 8504 ts_tablet_manager.cc:1939] T 062c542b638c41cab8e1591af6c5b2a2 P 2d89a41682c544d1a5d1bc58d6825287: Deleting consensus metadata
I20260924 07:11:48.139995 7909 catalog_manager.cc:4841] TS 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507): tablet 062c542b638c41cab8e1591af6c5b2a2 (table test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]) successfully deleted
I20260924 07:11:48.143148 8505 tablet_replica.cc:330] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: stopping tablet replica
I20260924 07:11:48.143426 8503 tablet_replica.cc:330] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: stopping tablet replica
I20260924 07:11:48.143633 8505 raft_consensus.cc:2229] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:48.143813 8503 raft_consensus.cc:2229] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.144223 8505 raft_consensus.cc:2258] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.144414 8503 raft_consensus.cc:2258] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.145156 8503 ts_tablet_manager.cc:1905] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.145855 8505 ts_tablet_manager.cc:1905] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.148790 8505 ts_tablet_manager.cc:1918] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:48.148862 8505 log.cc:1196] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/062c542b638c41cab8e1591af6c5b2a2
I20260924 07:11:48.149072 8505 ts_tablet_manager.cc:1939] T 062c542b638c41cab8e1591af6c5b2a2 P 2c26b440205b48eaa720cba94506a70d: Deleting consensus metadata
I20260924 07:11:48.149879 7912 catalog_manager.cc:4841] TS 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579): tablet 062c542b638c41cab8e1591af6c5b2a2 (table test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]) successfully deleted
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[7880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260924 07:11:48.152062 8503 ts_tablet_manager.cc:1918] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:48.152136 8503 log.cc:1196] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/062c542b638c41cab8e1591af6c5b2a2
I20260924 07:11:48.152361 8503 ts_tablet_manager.cc:1939] T 062c542b638c41cab8e1591af6c5b2a2 P 87e8b13414cc4976ba49215a3b1efddd: Deleting consensus metadata
I20260924 07:11:48.153168 7909 catalog_manager.cc:4841] TS 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549): tablet 062c542b638c41cab8e1591af6c5b2a2 (table test-table [id=293a2ba15a554e71b22b8ffb311e9c5a]) successfully deleted
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:11:48.168304 8513 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:48.160654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.0.98.62:38321)
0924 07:11:48.160923 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.160927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.160945 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:11:48.161082 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.161085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.161183 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.161280 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:48.162235 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.162843 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.163978 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.164308 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.164404 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.168095 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38014: BlockingRecv error: recv got EOF from 127.0.0.1:38014 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[7880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:11:48.179867 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.169762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.98.62:38321)
0924 07:11:48.169872 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.169876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.170234 (+ 358us) server_negotiation.cc:407] Connection header received
0924 07:11:48.170352 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.170355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.170400 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.170465 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:48.171195 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.172062 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.173193 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.173505 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.173572 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.176301 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:48.176305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:48.176308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:48.176338 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:48.178016 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:48.178490 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:48.178494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:48.178495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:48.178580 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:48.178839 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:48.178842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:48.178844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:48.179062 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:48.179181 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:48.179560 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 07:11:48.179696 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:11:48.183260 7925 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:48.183799 7925 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:48.188702 8062 tablet_service.cc:1467] Processing CreateTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf (DEFAULT_TABLE table=test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.188774 8342 tablet_service.cc:1467] Processing CreateTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf (DEFAULT_TABLE table=test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.188999 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee20f08777664a2bbfc6fae6ecdf7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.189049 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee20f08777664a2bbfc6fae6ecdf7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.191005 8519 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Bootstrap starting.
I20260924 07:11:48.191212 8186 tablet_service.cc:1467] Processing CreateTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf (DEFAULT_TABLE table=test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.191512 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee20f08777664a2bbfc6fae6ecdf7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.192368 8519 tablet_bootstrap.cc:654] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.193568 8519 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: No bootstrap required, opened a new log
I20260924 07:11:48.193671 8519 ts_tablet_manager.cc:1397] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:48.194109 8520 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Bootstrap starting.
I20260924 07:11:48.194222 8519 raft_consensus.cc:348] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.194321 8519 raft_consensus.cc:374] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.194350 8519 raft_consensus.cc:729] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.194494 8519 consensus_queue.cc:260] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.194777 8519 ts_tablet_manager.cc:1428] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:48.195027 8522 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Bootstrap starting.
I20260924 07:11:48.195185 8520 tablet_bootstrap.cc:654] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.195847 8522 tablet_bootstrap.cc:654] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.196931 8520 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: No bootstrap required, opened a new log
I20260924 07:11:48.197017 8520 ts_tablet_manager.cc:1397] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:48.197081 8522 tablet_bootstrap.cc:492] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: No bootstrap required, opened a new log
I20260924 07:11:48.197163 8522 ts_tablet_manager.cc:1397] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:11:48.197495 8522 raft_consensus.cc:348] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.197595 8522 raft_consensus.cc:374] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.197638 8522 raft_consensus.cc:729] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.197780 8522 consensus_queue.cc:260] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.197460 8520 raft_consensus.cc:348] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.198256 8520 raft_consensus.cc:374] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.198300 8520 raft_consensus.cc:729] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89a41682c544d1a5d1bc58d6825287, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.198441 8520 consensus_queue.cc:260] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.198714 8522 ts_tablet_manager.cc:1428] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:11:48.199105 8520 ts_tablet_manager.cc:1428] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:11:48.199958 8500 raft_consensus.cc:482] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:48.200040 8500 raft_consensus.cc:504] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.200374 8500 leader_election.cc:288] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:48.200764 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee20f08777664a2bbfc6fae6ecdf7acf" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287" is_pre_election: true
I20260924 07:11:48.200775 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee20f08777664a2bbfc6fae6ecdf7acf" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd" is_pre_election: true
I20260924 07:11:48.200882 8362 raft_consensus.cc:2454] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:48.200882 8081 raft_consensus.cc:2454] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:48.201154 8153 leader_election.cc:302] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 87e8b13414cc4976ba49215a3b1efddd; no voters:
I20260924 07:11:48.201292 8500 raft_consensus.cc:2781] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:48.201339 8500 raft_consensus.cc:482] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:48.201375 8500 raft_consensus.cc:3037] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.202294 8500 raft_consensus.cc:504] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.202600 8500 leader_election.cc:288] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Requested vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:48.202906 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee20f08777664a2bbfc6fae6ecdf7acf" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd"
I20260924 07:11:48.202944 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee20f08777664a2bbfc6fae6ecdf7acf" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287"
I20260924 07:11:48.203032 8081 raft_consensus.cc:3037] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.203042 8362 raft_consensus.cc:3037] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.203922 8362 raft_consensus.cc:2454] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:48.204192 8153 leader_election.cc:302] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 2d89a41682c544d1a5d1bc58d6825287; no voters:
I20260924 07:11:48.204329 8081 raft_consensus.cc:2454] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:48.204351 8500 raft_consensus.cc:2781] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:48.204423 8500 raft_consensus.cc:686] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Becoming Leader. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Running, Role: LEADER
I20260924 07:11:48.204566 8500 consensus_queue.cc:237] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.206102 7925 catalog_manager.cc:5495] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c26b440205b48eaa720cba94506a70d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:48.217504 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.214056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.0.98.2:43579)
0924 07:11:48.214181 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:48.214185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:48.214200 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:48.214300 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:48.214303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:48.214354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:48.214435 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:48.214618 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.215414 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.216371 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:48.216657 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:48.216758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:48.217002 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:48.217093 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:48.217159 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:48.217309 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:11:48.217364 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:11:48.219614 8362 raft_consensus.cc:1261] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.219652 8081 raft_consensus.cc:1261] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.220052 8500 consensus_queue.cc:1035] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.220233 8441 consensus_queue.cc:1035] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.227480 7923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.227716 7923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.229879 7923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Sending DeleteTablet for 3 replicas of tablet ee20f08777664a2bbfc6fae6ecdf7acf
I20260924 07:11:48.230552 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260924 07:11:48.230628 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38402
I20260924 07:11:48.230741 8503 tablet_replica.cc:330] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: stopping tablet replica
I20260924 07:11:48.230815 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet ee20f08777664a2bbfc6fae6ecdf7acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60662
I20260924 07:11:48.230863 8503 raft_consensus.cc:2229] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.230979 8505 tablet_replica.cc:330] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: stopping tablet replica
I20260924 07:11:48.231004 8503 raft_consensus.cc:2258] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.231091 8505 raft_consensus.cc:2229] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:48.231279 8505 raft_consensus.cc:2258] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.231211 7923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:11:48.231484 8504 tablet_replica.cc:330] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: stopping tablet replica
I20260924 07:11:48.231585 8504 raft_consensus.cc:2229] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.231709 8504 raft_consensus.cc:2258] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:11:48.231818 7923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:48.232492 8505 ts_tablet_manager.cc:1905] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.234926 8505 ts_tablet_manager.cc:1918] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:48.235001 8505 log.cc:1196] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/ee20f08777664a2bbfc6fae6ecdf7acf
I20260924 07:11:48.235229 8505 ts_tablet_manager.cc:1939] T ee20f08777664a2bbfc6fae6ecdf7acf P 2c26b440205b48eaa720cba94506a70d: Deleting consensus metadata
I20260924 07:11:48.235791 8504 ts_tablet_manager.cc:1905] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.236574 8062 tablet_service.cc:1467] Processing CreateTablet for tablet cce8eeaedf47486787366cf26c891e46 (DEFAULT_TABLE table=test-table [id=e2b41b2b4e1f40c69ef6cde9dd0a5d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.236867 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce8eeaedf47486787366cf26c891e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.238103 8504 ts_tablet_manager.cc:1918] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:48.238188 8504 log.cc:1196] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/ee20f08777664a2bbfc6fae6ecdf7acf
I20260924 07:11:48.238402 8504 ts_tablet_manager.cc:1939] T ee20f08777664a2bbfc6fae6ecdf7acf P 2d89a41682c544d1a5d1bc58d6825287: Deleting consensus metadata
I20260924 07:11:48.239081 8186 tablet_service.cc:1467] Processing CreateTablet for tablet cce8eeaedf47486787366cf26c891e46 (DEFAULT_TABLE table=test-table [id=e2b41b2b4e1f40c69ef6cde9dd0a5d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.239198 8519 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: Bootstrap starting.
I20260924 07:11:48.239331 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce8eeaedf47486787366cf26c891e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.240046 8519 tablet_bootstrap.cc:654] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.240897 8342 tablet_service.cc:1467] Processing CreateTablet for tablet cce8eeaedf47486787366cf26c891e46 (DEFAULT_TABLE table=test-table [id=e2b41b2b4e1f40c69ef6cde9dd0a5d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:48.241119 8342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce8eeaedf47486787366cf26c891e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:48.241654 8522 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Bootstrap starting.
I20260924 07:11:48.242110 8503 ts_tablet_manager.cc:1905] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.243496 8520 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Bootstrap starting.
I20260924 07:11:48.244082 8522 tablet_bootstrap.cc:654] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.244645 8503 ts_tablet_manager.cc:1918] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:48.244735 8503 log.cc:1196] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/ee20f08777664a2bbfc6fae6ecdf7acf
I20260924 07:11:48.244753 8520 tablet_bootstrap.cc:654] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:48.244962 8503 ts_tablet_manager.cc:1939] T ee20f08777664a2bbfc6fae6ecdf7acf P 87e8b13414cc4976ba49215a3b1efddd: Deleting consensus metadata
I20260924 07:11:48.245764 7909 catalog_manager.cc:4841] TS 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549): tablet ee20f08777664a2bbfc6fae6ecdf7acf (table test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]) successfully deleted
I20260924 07:11:48.246265 8519 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: No bootstrap required, opened a new log
I20260924 07:11:48.246351 8519 ts_tablet_manager.cc:1397] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 07:11:48.246232 8522 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: No bootstrap required, opened a new log
I20260924 07:11:48.246474 8522 ts_tablet_manager.cc:1397] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:11:48.246732 8519 raft_consensus.cc:348] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.246872 8522 raft_consensus.cc:348] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.246945 8519 raft_consensus.cc:374] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.246971 8522 raft_consensus.cc:374] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.247001 8522 raft_consensus.cc:729] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.247002 8519 raft_consensus.cc:729] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e8b13414cc4976ba49215a3b1efddd, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.247130 8519 consensus_queue.cc:260] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.247150 8522 consensus_queue.cc:260] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.247468 8522 ts_tablet_manager.cc:1428] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:48.247474 8519 ts_tablet_manager.cc:1428] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:48.248240 8520 tablet_bootstrap.cc:492] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: No bootstrap required, opened a new log
I20260924 07:11:48.248313 8520 ts_tablet_manager.cc:1397] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:11:48.248397 7909 catalog_manager.cc:4841] TS 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507): tablet ee20f08777664a2bbfc6fae6ecdf7acf (table test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]) successfully deleted
I20260924 07:11:48.248706 8520 raft_consensus.cc:348] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.248807 8520 raft_consensus.cc:374] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:48.248837 8520 raft_consensus.cc:729] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d89a41682c544d1a5d1bc58d6825287, State: Initialized, Role: FOLLOWER
I20260924 07:11:48.248943 8520 consensus_queue.cc:260] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.249051 7912 catalog_manager.cc:4841] TS 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579): tablet ee20f08777664a2bbfc6fae6ecdf7acf (table test-table [id=74fe19a1f45f42df98bb8b6e6181bb75]) successfully deleted
I20260924 07:11:48.249154 8520 ts_tablet_manager.cc:1428] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:48.304123 8500 raft_consensus.cc:482] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:48.304281 8500 raft_consensus.cc:504] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.304692 8500 leader_election.cc:288] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:48.305166 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce8eeaedf47486787366cf26c891e46" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287" is_pre_election: true
I20260924 07:11:48.305197 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce8eeaedf47486787366cf26c891e46" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd" is_pre_election: true
I20260924 07:11:48.305307 8362 raft_consensus.cc:2454] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:48.305377 8081 raft_consensus.cc:2454] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 0.
I20260924 07:11:48.305630 8153 leader_election.cc:302] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 2d89a41682c544d1a5d1bc58d6825287; no voters:
I20260924 07:11:48.305799 8500 raft_consensus.cc:2781] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:48.305860 8500 raft_consensus.cc:482] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:48.305905 8500 raft_consensus.cc:3037] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.306849 8500 raft_consensus.cc:504] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.307168 8500 leader_election.cc:288] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Requested vote from peers 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549), 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507)
I20260924 07:11:48.307600 8081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce8eeaedf47486787366cf26c891e46" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e8b13414cc4976ba49215a3b1efddd"
I20260924 07:11:48.307610 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce8eeaedf47486787366cf26c891e46" candidate_uuid: "2c26b440205b48eaa720cba94506a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d89a41682c544d1a5d1bc58d6825287"
I20260924 07:11:48.307722 8081 raft_consensus.cc:3037] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.307765 8362 raft_consensus.cc:3037] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:48.308622 8081 raft_consensus.cc:2454] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:48.308902 8362 raft_consensus.cc:2454] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c26b440205b48eaa720cba94506a70d in term 1.
I20260924 07:11:48.308943 8153 leader_election.cc:302] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c26b440205b48eaa720cba94506a70d, 87e8b13414cc4976ba49215a3b1efddd; no voters:
I20260924 07:11:48.309091 8500 raft_consensus.cc:2781] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:48.309175 8500 raft_consensus.cc:686] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Becoming Leader. State: Replica: 2c26b440205b48eaa720cba94506a70d, State: Running, Role: LEADER
I20260924 07:11:48.309312 8500 consensus_queue.cc:237] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } }
I20260924 07:11:48.311046 7924 catalog_manager.cc:5495] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c26b440205b48eaa720cba94506a70d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c26b440205b48eaa720cba94506a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c26b440205b48eaa720cba94506a70d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:48.330914 8362 raft_consensus.cc:1261] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.331030 8081 raft_consensus.cc:1261] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Refusing update from remote peer 2c26b440205b48eaa720cba94506a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:48.331310 8467 consensus_queue.cc:1035] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d89a41682c544d1a5d1bc58d6825287" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.331516 8500 consensus_queue.cc:1035] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e8b13414cc4976ba49215a3b1efddd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:48.362519 7924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.362655 7924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:48.364547 7924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db5ca48dfa6546e5a8f887b195e498ca: Sending DeleteTablet for 3 replicas of tablet cce8eeaedf47486787366cf26c891e46
I20260924 07:11:48.365125 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet cce8eeaedf47486787366cf26c891e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38402
I20260924 07:11:48.365161 8342 tablet_service.cc:1514] Processing DeleteTablet for tablet cce8eeaedf47486787366cf26c891e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60662
I20260924 07:11:48.365161 8062 tablet_service.cc:1514] Processing DeleteTablet for tablet cce8eeaedf47486787366cf26c891e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47742
I20260924 07:11:48.365351 8505 tablet_replica.cc:330] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: stopping tablet replica
I20260924 07:11:48.365396 8503 tablet_replica.cc:330] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd: stopping tablet replica
I20260924 07:11:48.365481 8505 raft_consensus.cc:2229] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:48.365528 8503 raft_consensus.cc:2229] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.365645 8503 raft_consensus.cc:2258] T cce8eeaedf47486787366cf26c891e46 P 87e8b13414cc4976ba49215a3b1efddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.365643 8505 raft_consensus.cc:2258] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.365760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7997
I20260924 07:11:48.365833 8504 tablet_replica.cc:330] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: stopping tablet replica
I20260924 07:11:48.365936 8504 raft_consensus.cc:2229] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:48.366030 8504 raft_consensus.cc:2258] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:48.366762 8504 ts_tablet_manager.cc:1905] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.367626 8505 ts_tablet_manager.cc:1905] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:48.369055 8504 ts_tablet_manager.cc:1918] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:48.369125 8504 log.cc:1196] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/cce8eeaedf47486787366cf26c891e46
I20260924 07:11:48.369334 8504 ts_tablet_manager.cc:1939] T cce8eeaedf47486787366cf26c891e46 P 2d89a41682c544d1a5d1bc58d6825287: Deleting consensus metadata
I20260924 07:11:48.370085 7909 catalog_manager.cc:4841] TS 2d89a41682c544d1a5d1bc58d6825287 (127.0.98.3:35507): tablet cce8eeaedf47486787366cf26c891e46 (table test-table [id=e2b41b2b4e1f40c69ef6cde9dd0a5d02]) successfully deleted
I20260924 07:11:48.370139 8505 ts_tablet_manager.cc:1918] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:11:48.370205 8505 log.cc:1196] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/cce8eeaedf47486787366cf26c891e46
I20260924 07:11:48.370429 8505 ts_tablet_manager.cc:1939] T cce8eeaedf47486787366cf26c891e46 P 2c26b440205b48eaa720cba94506a70d: Deleting consensus metadata
I20260924 07:11:48.371057 7912 catalog_manager.cc:4841] TS 2c26b440205b48eaa720cba94506a70d (127.0.98.2:43579): tablet cce8eeaedf47486787366cf26c891e46 (table test-table [id=e2b41b2b4e1f40c69ef6cde9dd0a5d02]) successfully deleted
W20260924 07:11:48.373647 7909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43549 (error 108)
I20260924 07:11:48.373777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8137
I20260924 07:11:48.375571 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:48.375176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43549 (local address 127.0.0.1:47746)
0924 07:11:48.375416 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:48.375486 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
W20260924 07:11:48.375707 7909 catalog_manager.cc:4584] TS 87e8b13414cc4976ba49215a3b1efddd (127.0.98.1:43549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cce8eeaedf47486787366cf26c891e46: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43549: connect: Connection refused (error 111)
I20260924 07:11:48.379930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8277
I20260924 07:11:48.385427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 7893
2026-09-24T07:11:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3154 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[8539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[8539](info): set up 2 sockets
Sep 24 07:11:48 dist-test-slave-cnv3 krb5kdc[8539](info): commencing operation
krb5kdc: starting...
W20260924 07:11:49.343850 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:49 dist-test-slave-cnv3 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:49.467485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:49.577407 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:49.577706 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:49.577749 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:49.582084 8552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:49.582161 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:49.582180 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:49.582196 8552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:49.582211 8552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:49.587877 8552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:49.589018 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:49.590070 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:49.598405 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:49.598338 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:49.598347 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:49.598701 8552 server_base.cc:1034] running on GCE node
I20260924 07:11:49.599272 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:49.600358 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:49.601517 8552 hybrid_clock.cc:648] HybridClock initialized: now 1790233909601499 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T07:11:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1220 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:49 dist-test-slave-cnv3 krb5kdc[8565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:49 dist-test-slave-cnv3 krb5kdc[8565](info): set up 2 sockets
Sep 24 07:11:49 dist-test-slave-cnv3 krb5kdc[8565](info): commencing operation
krb5kdc: starting...
W20260924 07:11:50.567864 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:50 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:50.691259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:50.803021 8578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:50.803313 8578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:50.803352 8578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:50.807524 8578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:50.807693 8578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:50.807713 8578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:50.807729 8578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:50.807744 8578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:50.813216 8578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:50.814311 8578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:50.815347 8578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:50.823925 8584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:50.824018 8578 server_base.cc:1034] running on GCE node
W20260924 07:11:50.823908 8586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:50.823902 8583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:50.824750 8578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:50.826049 8578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:50.827225 8578 hybrid_clock.cc:648] HybridClock initialized: now 1790233910827206 us; error 35 us; skew 500 ppm
Sep 24 07:11:50 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:50.830066 8578 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:50.830973 8578 webserver.cc:460] Webserver started at http://127.0.98.62:36085/ using document root <none> and password file <none>
I20260924 07:11:50.831561 8578 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:50.831625 8578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:50.831802 8578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:50.833895 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "6e4aea76943f4ed1bb34fa8771db8689"
format_stamp: "Formatted at 2026-09-24 07:11:50 on dist-test-slave-cnv3"
server_key: "e54ae61faca5016398d8b343ae08a8d8"
server_key_iv: "90bc9dce79fb2833514a57b6b17a232c"
server_key_version: "encryptionkey@0"
I20260924 07:11:50.834427 8578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e4aea76943f4ed1bb34fa8771db8689"
format_stamp: "Formatted at 2026-09-24 07:11:50 on dist-test-slave-cnv3"
server_key: "e54ae61faca5016398d8b343ae08a8d8"
server_key_iv: "90bc9dce79fb2833514a57b6b17a232c"
server_key_version: "encryptionkey@0"
I20260924 07:11:50.838333 8578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:11:50.841032 8593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:50.842455 8578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:50.842585 8578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "6e4aea76943f4ed1bb34fa8771db8689"
format_stamp: "Formatted at 2026-09-24 07:11:50 on dist-test-slave-cnv3"
server_key: "e54ae61faca5016398d8b343ae08a8d8"
server_key_iv: "90bc9dce79fb2833514a57b6b17a232c"
server_key_version: "encryptionkey@0"
I20260924 07:11:50.842700 8578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:50.868289 8578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:50.871464 8578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:50.871637 8578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:50.878263 8578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40397
I20260924 07:11:50.878281 8645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40397 every 8 connection(s)
I20260924 07:11:50.879197 8578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:50.881930 8646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:50.887424 8646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Bootstrap starting.
I20260924 07:11:50.887756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8578
I20260924 07:11:50.887866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:50.888108 392 external_mini_cluster.cc:1428] Setting key cf60cc35868f2b49b2f29969842282f2
I20260924 07:11:50.889976 8646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:50.890726 8646 log.cc:824] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:50.892939 8646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: No bootstrap required, opened a new log
Sep 24 07:11:50 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:50.897172 8646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:50.897423 8646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:50.897487 8646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4aea76943f4ed1bb34fa8771db8689, State: Initialized, Role: FOLLOWER
I20260924 07:11:50.897984 8646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:50.898140 8646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:50.898255 8646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:50.898366 8646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:50.899497 8646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:50.899859 8646 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4aea76943f4ed1bb34fa8771db8689; no voters:
I20260924 07:11:50.900151 8646 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:50.900231 8651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:50.900389 8651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 LEADER]: Becoming Leader. State: Replica: 6e4aea76943f4ed1bb34fa8771db8689, State: Running, Role: LEADER
I20260924 07:11:50.900676 8651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:50.901424 8646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:50.902333 8652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4aea76943f4ed1bb34fa8771db8689. Latest consensus state: current_term: 1 leader_uuid: "6e4aea76943f4ed1bb34fa8771db8689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260924 07:11:50.902455 8652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:50.902369 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e4aea76943f4ed1bb34fa8771db8689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260924 07:11:50.902621 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:50.902823 8660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:50.902854 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:50.889391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.98.62:40397)
0924 07:11:50.889799 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:50.889813 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:50.889851 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:11:50.890536 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:50.890554 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:50.890948 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:50.891302 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:50.892210 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:50.893279 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:50.894410 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:50.894865 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:50.894995 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:50.897700 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:50.897707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:50.897723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:50.897753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:50.899668 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:50.900078 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:50.900085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:50.900090 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:50.900209 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:50.900516 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:50.900519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:50.900521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:50.901066 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:50.901267 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:50.901551 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 07:11:50.901861 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":102,"threads_started":1}
I20260924 07:11:50.905787 8660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:50.911204 8660 catalog_manager.cc:1347] Generated new cluster ID: 5e1716faaf0b43a8b82bbf8c162f0543
I20260924 07:11:50.911263 8660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:50.916131 8660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:50.916988 8660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:50.925462 8660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Generated new TSK 0
I20260924 07:11:50.926283 8660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:50.955230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:51.069006 8672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:51.069278 8672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:51.069321 8672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:51.073881 8672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:51.073958 8672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:51.074023 8672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:51.079412 8672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:51.080480 8672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:51.081549 8672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:51.091325 8677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.091398 8672 server_base.cc:1034] running on GCE node
W20260924 07:11:51.091318 8680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.091318 8678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.091993 8672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:51.092669 8672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:51.093840 8672 hybrid_clock.cc:648] HybridClock initialized: now 1790233911093800 us; error 57 us; skew 500 ppm
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:51.096694 8672 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:51.097606 8672 webserver.cc:460] Webserver started at http://127.0.98.1:43857/ using document root <none> and password file <none>
I20260924 07:11:51.098165 8672 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:51.098232 8672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:51.098418 8672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:51.100600 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a503a01f1894b18a3a0234427b72d2b"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "137b2a6f8b7f095d84e820f53732562a"
server_key_iv: "a99cc1e045db2652a243a19438d9107e"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.101215 8672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a503a01f1894b18a3a0234427b72d2b"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "137b2a6f8b7f095d84e820f53732562a"
server_key_iv: "a99cc1e045db2652a243a19438d9107e"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.105134 8672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:11:51.107677 8687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.108996 8672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:51.109117 8672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "9a503a01f1894b18a3a0234427b72d2b"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "137b2a6f8b7f095d84e820f53732562a"
server_key_iv: "a99cc1e045db2652a243a19438d9107e"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.109210 8672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:51.122836 8672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:51.125976 8672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:51.126153 8672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:51.126706 8672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:51.127576 8672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:51.127630 8672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.127671 8672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:51.127696 8672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.136241 8672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36319
I20260924 07:11:51.136270 8800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36319 every 8 connection(s)
I20260924 07:11:51.137132 8672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:51.141794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8672
I20260924 07:11:51.141916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:51.142174 392 external_mini_cluster.cc:1428] Setting key 39510045a1552377aec20adf1d187c00
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:51.151702 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.138963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57605 (local address 127.0.98.62:40397)
0924 07:11:51.139160 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:51.139165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:51.140052 (+ 887us) server_negotiation.cc:407] Connection header received
0924 07:11:51.141517 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:51.141520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:51.141578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:51.141657 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:51.143176 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.143862 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.145009 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.145235 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.145328 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.148138 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:51.148142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:51.148145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:51.148179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:51.149659 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.150172 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.150175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.150177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.150255 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.150585 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.150589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.150590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.150768 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:51.150850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:51.151398 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 07:11:51.151523 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:11:51.152597 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.139405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:57605)
0924 07:11:51.139899 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:51.139941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:51.141290 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:51.141841 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:51.141848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:51.142273 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:51.142994 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.143012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.143984 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.143986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.144894 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.144903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.145401 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.145405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.145759 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.146390 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:51.146406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:51.147973 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:51.149775 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.149780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.149797 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.150080 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.150342 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.150345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.150347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.150497 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.150864 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:51.150871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:51.151128 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:11:51.151339 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 07:11:51.151695 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:11:51.153860 8801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40397
I20260924 07:11:51.154114 8801 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:51.154600 8801 heartbeater.cc:511] Master 127.0.98.62:40397 requested a full tablet report, sending...
I20260924 07:11:51.156059 8610 ts_manager.cc:194] Registered new tserver with Master: 9a503a01f1894b18a3a0234427b72d2b (127.0.98.1:36319)
I20260924 07:11:51.157173 8610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:51.170364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:51.280786 8806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:51.281066 8806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:51.281109 8806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:51.285240 8806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:51.285317 8806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:51.285380 8806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:51.290836 8806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:51.292017 8806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:51.293040 8806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:51.303056 8812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.303035 8814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.303021 8811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.303434 8806 server_base.cc:1034] running on GCE node
I20260924 07:11:51.303848 8806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:51.304489 8806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:51.305662 8806 hybrid_clock.cc:648] HybridClock initialized: now 1790233911305625 us; error 50 us; skew 500 ppm
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:51.308403 8806 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:51.309312 8806 webserver.cc:460] Webserver started at http://127.0.98.2:35979/ using document root <none> and password file <none>
I20260924 07:11:51.309873 8806 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:51.309942 8806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:51.310130 8806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:51.312270 8806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f19babe38ba4f2c8caf57c1bc119dc1"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "6eaad3ba05c981f68d11db7bcb10ef8b"
server_key_iv: "5463dcbed6355b1c2150c78ad5f8ad40"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.312819 8806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f19babe38ba4f2c8caf57c1bc119dc1"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "6eaad3ba05c981f68d11db7bcb10ef8b"
server_key_iv: "5463dcbed6355b1c2150c78ad5f8ad40"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.316699 8806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:11:51.319267 8821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.320569 8806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 07:11:51.320690 8806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "0f19babe38ba4f2c8caf57c1bc119dc1"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "6eaad3ba05c981f68d11db7bcb10ef8b"
server_key_iv: "5463dcbed6355b1c2150c78ad5f8ad40"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.320791 8806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:51.335297 8806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:51.338550 8806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:51.338743 8806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:51.339341 8806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:51.340204 8806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:51.340250 8806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.340306 8806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:51.340334 8806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.348903 8806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40077
I20260924 07:11:51.348969 8934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40077 every 8 connection(s)
I20260924 07:11:51.349843 8806 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:51.356870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8806
I20260924 07:11:51.357007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:51.357311 392 external_mini_cluster.cc:1428] Setting key 4480f9902fe3abdca73bf151e13ac5a1
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:51.364493 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.351782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44781 (local address 127.0.98.62:40397)
0924 07:11:51.351921 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:51.351925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:51.352752 (+ 827us) server_negotiation.cc:407] Connection header received
0924 07:11:51.354157 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:51.354161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:51.354209 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:51.354285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:51.355592 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.356120 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.357735 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.358013 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.358095 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.360724 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:51.360728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:51.360730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:51.360758 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:51.362293 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.362900 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.362904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.362906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.362995 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.363318 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.363322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.363323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.363574 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:51.363689 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:51.364199 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 07:11:51.364326 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:11:51.365334 8937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.352145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:44781)
0924 07:11:51.352594 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:51.352640 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:51.353930 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:51.354448 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:51.354457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:51.354894 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:51.355447 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.355464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.356227 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.356230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.357579 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.357591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.358177 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.358180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.358350 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.358912 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:51.358933 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:51.360542 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:51.362404 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.362409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.362422 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.362710 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.363089 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.363092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.363093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.363233 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.363683 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:51.363689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:51.363936 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 07:11:51.364132 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 07:11:51.364414 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260924 07:11:51.366546 8935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40397
I20260924 07:11:51.366786 8935 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:51.367274 8935 heartbeater.cc:511] Master 127.0.98.62:40397 requested a full tablet report, sending...
I20260924 07:11:51.368445 8610 ts_manager.cc:194] Registered new tserver with Master: 0f19babe38ba4f2c8caf57c1bc119dc1 (127.0.98.2:40077)
I20260924 07:11:51.369040 8610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:51.387099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:51.501386 8940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:51.501649 8940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:51.501693 8940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:51.505913 8940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:51.505982 8940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:51.506044 8940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:51.511418 8940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:51.512457 8940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:51.513485 8940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:51.523335 8948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.523325 8946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.523325 8945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.523594 8940 server_base.cc:1034] running on GCE node
I20260924 07:11:51.524179 8940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:51.524806 8940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:51.526005 8940 hybrid_clock.cc:648] HybridClock initialized: now 1790233911525964 us; error 65 us; skew 500 ppm
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:51.528812 8940 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:51.529678 8940 webserver.cc:460] Webserver started at http://127.0.98.3:40651/ using document root <none> and password file <none>
I20260924 07:11:51.530205 8940 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:51.530263 8940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:51.530424 8940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:11:51.532495 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "c1ebcd6334484815a3d94359e6779ae0"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "ece7cb70aa351688e5646c17584e55da"
server_key_iv: "90213201d98741da2713cb69d49dc101"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.533043 8940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c1ebcd6334484815a3d94359e6779ae0"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "ece7cb70aa351688e5646c17584e55da"
server_key_iv: "90213201d98741da2713cb69d49dc101"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.536731 8940 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:51.539253 8955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.540611 8940 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:51.540709 8940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "c1ebcd6334484815a3d94359e6779ae0"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "ece7cb70aa351688e5646c17584e55da"
server_key_iv: "90213201d98741da2713cb69d49dc101"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.540802 8940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:51.553475 8940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:51.556749 8940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:51.556910 8940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:51.557458 8940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:51.558251 8940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:51.558295 8940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.558342 8940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:51.558362 8940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.566705 8940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37115
I20260924 07:11:51.566735 9068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37115 every 8 connection(s)
I20260924 07:11:51.567638 8940 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:51.573491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 8940
I20260924 07:11:51.573589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:51.573844 392 external_mini_cluster.cc:1428] Setting key c6cde15a801f3ca2cf4e463d72647ff0
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:51.581029 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.569461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46803 (local address 127.0.98.62:40397)
0924 07:11:51.569718 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:51.569723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:51.570499 (+ 776us) server_negotiation.cc:407] Connection header received
0924 07:11:51.571923 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:51.571926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:51.571980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:51.572056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:51.573301 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.573894 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.575057 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.575286 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.575394 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.577844 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:51.577847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:51.577849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:51.577875 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:51.579151 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.579694 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.579697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.579698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.579772 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:51.580045 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:51.580047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:51.580048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:51.580219 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:51.580330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:51.580804 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 07:11:51.580907 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 07:11:51.581818 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:51.569891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:46803)
0924 07:11:51.570349 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:51.570393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:51.571693 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:51.572216 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:51.572224 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:51.572634 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:51.573139 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.573153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.574020 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.574023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.574936 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:51.574946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:51.575449 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:51.575454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:51.575638 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:51.576145 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:51.576161 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:51.577671 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:51.579276 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.579280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.579295 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.579603 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.579842 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:51.579844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:51.579845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:51.579979 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:51.580310 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:51.580315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:51.580558 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 07:11:51.580727 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:11:51.580994 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":106,"threads_started":1}
I20260924 07:11:51.582998 9069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40397
I20260924 07:11:51.583231 9069 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:51.583782 9069 heartbeater.cc:511] Master 127.0.98.62:40397 requested a full tablet report, sending...
I20260924 07:11:51.584841 8610 ts_manager.cc:194] Registered new tserver with Master: c1ebcd6334484815a3d94359e6779ae0 (127.0.98.3:37115)
I20260924 07:11:51.585397 8610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46803
I20260924 07:11:51.587500 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:11:51.662513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8672
I20260924 07:11:51.667507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8806
I20260924 07:11:51.672180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8940
I20260924 07:11:51.676597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 8578
I20260924 07:11:51.682116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--webserver_interface=127.0.98.62
--webserver_port=36085
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:51.791527 9078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:51.791813 9078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:51.791855 9078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:51.795989 9078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:51.796072 9078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:51.796090 9078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:51.796106 9078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:51.796120 9078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:51.801607 9078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:51.802755 9078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:51.803869 9078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:51.812214 9083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.812212 9084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.812361 9078 server_base.cc:1034] running on GCE node
W20260924 07:11:51.812183 9086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.813109 9078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:51.814237 9078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:51.815424 9078 hybrid_clock.cc:648] HybridClock initialized: now 1790233911815402 us; error 35 us; skew 500 ppm
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:51.818171 9078 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:51.819077 9078 webserver.cc:460] Webserver started at http://127.0.98.62:36085/ using document root <none> and password file <none>
I20260924 07:11:51.819670 9078 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:51.819754 9078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:51.823642 9078 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:51.825764 9093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:51.827001 9078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:51.827111 9078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "6e4aea76943f4ed1bb34fa8771db8689"
format_stamp: "Formatted at 2026-09-24 07:11:50 on dist-test-slave-cnv3"
server_key: "e54ae61faca5016398d8b343ae08a8d8"
server_key_iv: "90bc9dce79fb2833514a57b6b17a232c"
server_key_version: "encryptionkey@0"
I20260924 07:11:51.827682 9078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:51.839885 9078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:51.842672 9078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:51.842846 9078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:51.849839 9078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40397
I20260924 07:11:51.849900 9145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40397 every 8 connection(s)
I20260924 07:11:51.850941 9078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:51.855209 9146 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:11:51.856876 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Bootstrap starting.
I20260924 07:11:51.858242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9078
I20260924 07:11:51.858683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36319
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40397
--webserver_port=43857
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:11:51.873816 9146 log.cc:824] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:51.880550 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:11:51.880906 9146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Bootstrap complete.
I20260924 07:11:51.885082 9146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:51.885502 9146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4aea76943f4ed1bb34fa8771db8689, State: Initialized, Role: FOLLOWER
I20260924 07:11:51.885996 9146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:51.886116 9146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:51.886179 9146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:51.886267 9146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:11:51.888300 9146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:51.888640 9146 leader_election.cc:302] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e4aea76943f4ed1bb34fa8771db8689; no voters:
I20260924 07:11:51.888926 9146 leader_election.cc:288] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:11:51.889070 9150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:11:51.889339 9150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [term 2 LEADER]: Becoming Leader. State: Replica: 6e4aea76943f4ed1bb34fa8771db8689, State: Running, Role: LEADER
I20260924 07:11:51.889669 9150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260924 07:11:51.890148 9146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:51.891109 9151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6e4aea76943f4ed1bb34fa8771db8689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260924 07:11:51.891234 9151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:51.892081 9152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e4aea76943f4ed1bb34fa8771db8689. Latest consensus state: current_term: 2 leader_uuid: "6e4aea76943f4ed1bb34fa8771db8689" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e4aea76943f4ed1bb34fa8771db8689" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260924 07:11:51.892191 9152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:51.894771 9166 catalog_manager.cc:1259] Loaded cluster ID: 5e1716faaf0b43a8b82bbf8c162f0543
I20260924 07:11:51.894833 9166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: loading cluster ID for follower catalog manager: success
I20260924 07:11:51.896586 9166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: acquiring CA information for follower catalog manager: success
I20260924 07:11:51.897682 9166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:11:51.897797 9159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:51.898633 9159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:51.899049 9159 catalog_manager.cc:1259] Loaded cluster ID: 5e1716faaf0b43a8b82bbf8c162f0543
I20260924 07:11:51.899102 9159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:51.899701 9159 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:51.900110 9159 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6e4aea76943f4ed1bb34fa8771db8689: Loaded TSK: 0
I20260924 07:11:51.900803 9159 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:11:51.971472 9148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:51.971736 9148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:51.971782 9148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:51.975868 9148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:51.975929 9148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:51.975992 9148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:51.981267 9148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43857
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:51.982304 9148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:51.983289 9148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:51.992942 9173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.992966 9172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:51.993218 9175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:51.993127 9148 server_base.cc:1034] running on GCE node
I20260924 07:11:51.993614 9148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:51.994268 9148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:51.995466 9148 hybrid_clock.cc:648] HybridClock initialized: now 1790233911995425 us; error 54 us; skew 500 ppm
Sep 24 07:11:51 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233911, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:51.998469 9148 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:11:51.999552 9148 webserver.cc:460] Webserver started at http://127.0.98.1:43857/ using document root <none> and password file <none>
I20260924 07:11:52.000106 9148 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:52.000191 9148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:52.004097 9148 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:52.006191 9182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.007474 9148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:52.007572 9148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "9a503a01f1894b18a3a0234427b72d2b"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "137b2a6f8b7f095d84e820f53732562a"
server_key_iv: "a99cc1e045db2652a243a19438d9107e"
server_key_version: "encryptionkey@0"
I20260924 07:11:52.008129 9148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:52.054816 9148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:52.057989 9148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:52.058182 9148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:52.058792 9148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:52.059670 9148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:52.059718 9148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.059772 9148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:52.059800 9148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.068342 9148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36319
I20260924 07:11:52.068392 9295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36319 every 8 connection(s)
I20260924 07:11:52.069357 9148 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:52.075670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9148
I20260924 07:11:52.076138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40397
--webserver_port=35979
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:11:52.082891 9298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.071334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48301 (local address 127.0.98.62:40397)
0924 07:11:52.071825 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.071845 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.072151 (+ 306us) server_negotiation.cc:407] Connection header received
0924 07:11:52.073616 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.073629 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.073934 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.074206 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.075289 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.076838 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.078052 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.078525 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.078733 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.082124 (+ 3391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48301: BlockingRecv error: recv got EOF from 127.0.98.1:48301 (error 108)
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260924 07:11:52.082897 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.071527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:48301)
0924 07:11:52.072000 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.072042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.073226 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.074286 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.074292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.074651 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.075102 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.075115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.076990 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.076993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.077911 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.077921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.078685 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.078690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.078883 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.079551 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.079568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.081857 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
W20260924 07:11:52.083659 9296 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40397 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40397: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.089612 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.084136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:35265)
0924 07:11:52.084290 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.084307 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.084403 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.084655 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.084658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.084844 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.084972 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.084979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.085712 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.085715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.086640 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.086650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.087068 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.087071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.087165 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.087672 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.087687 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.089505 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260924 07:11:52.090423 9298 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.084259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35265 (local address 127.0.98.62:40397)
0924 07:11:52.084373 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.084376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.084390 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:11:52.084498 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.084499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.084540 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.084615 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.085076 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.085606 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.086756 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.086970 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.087046 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.089682 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35265: BlockingRecv error: recv got EOF from 127.0.98.1:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.095497 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.090237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:47599)
0924 07:11:52.090614 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.090628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.090709 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.091004 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.091007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.091199 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.091275 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.091280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.092083 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.092087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.092720 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.092730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.093169 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.093173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.093245 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.093688 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.093702 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.095346 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298}
W20260924 07:11:52.095629 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.090347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47599 (local address 127.0.98.62:40397)
0924 07:11:52.090675 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.090684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.090697 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:52.090799 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.090802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.090896 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.090972 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.091396 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.091980 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.092842 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.093069 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.093154 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.095508 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47599: BlockingRecv error: recv got EOF from 127.0.98.1:47599 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":145,"threads_started":1}
W20260924 07:11:52.095930 9296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:11:52.189738 9300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:52.190016 9300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:52.190075 9300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:52.194343 9300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:52.194415 9300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:52.194499 9300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:52.199994 9300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35979
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:52.201092 9300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:52.202193 9300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:11:52.211797 9300 server_base.cc:1034] running on GCE node
W20260924 07:11:52.211710 9307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:52.211719 9309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:52.211719 9306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:52.212422 9300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:52.213065 9300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:52.214242 9300 hybrid_clock.cc:648] HybridClock initialized: now 1790233912214216 us; error 38 us; skew 500 ppm
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:52.216964 9300 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:11:52.217892 9300 webserver.cc:460] Webserver started at http://127.0.98.2:35979/ using document root <none> and password file <none>
I20260924 07:11:52.218446 9300 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:52.218526 9300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:52.222486 9300 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:52.224725 9316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.226131 9300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:11:52.226256 9300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "0f19babe38ba4f2c8caf57c1bc119dc1"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "6eaad3ba05c981f68d11db7bcb10ef8b"
server_key_iv: "5463dcbed6355b1c2150c78ad5f8ad40"
server_key_version: "encryptionkey@0"
I20260924 07:11:52.226820 9300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:52.242961 9300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:52.246124 9300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:52.246310 9300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:52.246968 9300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:52.247834 9300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:52.247882 9300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.247936 9300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:52.247964 9300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.256851 9300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40077
I20260924 07:11:52.256878 9429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40077 every 8 connection(s)
I20260924 07:11:52.257886 9300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:52.262712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9300
I20260924 07:11:52.263177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37115
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40397
--webserver_port=40651
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:11:52.269672 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.259693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34813 (local address 127.0.98.62:40397)
0924 07:11:52.259864 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.259869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.260786 (+ 917us) server_negotiation.cc:407] Connection header received
0924 07:11:52.262087 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.262090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.262138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.262210 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.263829 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.264398 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.265568 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.265835 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.265933 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.269548 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34813: BlockingRecv error: recv got EOF from 127.0.98.2:34813 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:11:52.270329 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.260190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:34813)
0924 07:11:52.260635 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.260678 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.261883 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.262346 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.262353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.262857 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.263653 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.263675 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.264541 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.264544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.265451 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.265461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.265987 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.265991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.266171 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.266804 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.266820 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.269397 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
W20260924 07:11:52.270927 9430 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40397 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40397: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.276820 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.271418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:56771)
0924 07:11:52.271531 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.271545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.271625 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.271901 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.271904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.272045 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.272143 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.272148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.273187 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.273189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.273744 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.273753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.274298 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.274300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.274370 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.274893 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.274906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.276699 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:11:52.276965 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.271510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56771 (local address 127.0.98.62:40397)
0924 07:11:52.271637 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.271642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.271659 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:11:52.271704 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.271707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.271760 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.271851 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.272258 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.273050 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.273847 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.274194 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.274286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.276853 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56771: BlockingRecv error: recv got EOF from 127.0.98.2:56771 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.282100 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.277448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:44589)
0924 07:11:52.277598 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.277611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.277698 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.277908 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.277911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.278062 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.278128 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.278133 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.278798 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.278801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.279305 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.279312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.279731 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.279735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.279809 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.280253 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.280262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.281980 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260924 07:11:52.282326 9430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:11:52.282449 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.277521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44589 (local address 127.0.98.62:40397)
0924 07:11:52.277620 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.277624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.277652 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:11:52.277783 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.277786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.277826 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.277886 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.278233 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.278704 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.279423 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.279629 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.279710 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.282112 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44589: BlockingRecv error: recv got EOF from 127.0.98.2:44589 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:11:52.376226 9433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:52.376487 9433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:52.376546 9433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:52.380519 9433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:52.380576 9433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:52.380650 9433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:52.385891 9433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40651
--tserver_master_addrs=127.0.98.62:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:11:52.386947 9433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:52.388032 9433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:52.398267 9441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:52.398262 9438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:52.398262 9439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:52.398506 9433 server_base.cc:1034] running on GCE node
I20260924 07:11:52.398897 9433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:52.399631 9433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:52.400822 9433 hybrid_clock.cc:648] HybridClock initialized: now 1790233912400764 us; error 67 us; skew 500 ppm
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233912, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:52.403581 9433 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:11:52.404487 9433 webserver.cc:460] Webserver started at http://127.0.98.3:40651/ using document root <none> and password file <none>
I20260924 07:11:52.405014 9433 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:52.405092 9433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:52.409050 9433 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:11:52.411211 9448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.412578 9433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:11:52.412680 9433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "c1ebcd6334484815a3d94359e6779ae0"
format_stamp: "Formatted at 2026-09-24 07:11:51 on dist-test-slave-cnv3"
server_key: "ece7cb70aa351688e5646c17584e55da"
server_key_iv: "90213201d98741da2713cb69d49dc101"
server_key_version: "encryptionkey@0"
I20260924 07:11:52.413260 9433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:52.446208 9433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:52.449522 9433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:52.449700 9433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:52.450297 9433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:52.451145 9433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:52.451192 9433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.451232 9433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:52.451259 9433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:52.460474 9433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37115
I20260924 07:11:52.460541 9561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37115 every 8 connection(s)
I20260924 07:11:52.461504 9433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:52.470173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9433
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:11:52.472682 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.463408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38365 (local address 127.0.98.62:40397)
0924 07:11:52.463563 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.463567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.464422 (+ 855us) server_negotiation.cc:407] Connection header received
0924 07:11:52.465738 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.465742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.465789 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.465867 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.467084 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.467651 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.468859 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.469075 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.469158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.472563 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38365: BlockingRecv error: recv got EOF from 127.0.98.3:38365 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:11:52.473302 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.463811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:38365)
0924 07:11:52.464275 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.464318 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.465516 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.466072 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.466080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.466464 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.466953 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.466966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.467747 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.467749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.468735 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.468744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.469196 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.469199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.469337 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.469930 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.469946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.472436 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
W20260924 07:11:52.473757 9562 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40397 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40397: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.479137 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.474160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:37737)
0924 07:11:52.474270 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.474284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.474351 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.474541 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.474544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.474674 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.474762 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.474769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.475702 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.475704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.476276 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.476285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.476819 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.476823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.476915 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.477272 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.477283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.479015 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:11:52.479228 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.474220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37737 (local address 127.0.98.62:40397)
0924 07:11:52.474335 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.474338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.474350 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:52.474421 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.474424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.474460 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.474517 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.474875 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.475604 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.476376 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.476700 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.476804 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.479134 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37737: BlockingRecv error: recv got EOF from 127.0.98.3:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:11:52.479962 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.471162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.98.62:40397)
0924 07:11:52.471293 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.471297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.471310 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:52.471347 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.471350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.471421 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.471487 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.472332 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.473019 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.474129 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.474406 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.474492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.475954 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:52.475959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:52.475974 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:52.475999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:52.477637 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:52.478054 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:52.478061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:52.478068 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:52.478255 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:52.478544 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:52.478546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:52.478547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:52.478902 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:52.479094 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:52.479608 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:11:52.479831 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:52 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:52.484637 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:52.479727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:48331)
0924 07:11:52.479826 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:52.479839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:52.479905 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:52.480265 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:52.480268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:52.480424 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:52.480506 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.480512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.481391 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.481393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.482053 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:52.482062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.482544 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.482547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:52.482620 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.483036 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:52.483050 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:52.484552 (+ 1502us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:11:52.484865 9301 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:52.479880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48331 (local address 127.0.98.62:40397)
0924 07:11:52.480024 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:52.480028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:52.480039 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:52.480069 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:52.480072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:52.480107 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:52.480170 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:52.480618 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.481284 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.482158 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:52.482441 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:52.482547 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:52.484766 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48331: BlockingRecv error: recv got EOF from 127.0.98.3:48331 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 07:11:52.484862 9562 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:53.101961 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:53.096592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:59687)
0924 07:11:53.096861 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:53.096874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:53.096969 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:53.097258 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:53.097261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:53.097432 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:53.097534 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.097541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.098381 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.098383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.099003 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.099012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.099483 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.099486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.099574 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.100092 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:53.100104 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:53.101833 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260924 07:11:53.102108 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:53.096647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59687 (local address 127.0.98.62:40397)
0924 07:11:53.097022 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:53.097025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:53.097038 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:53.097073 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:53.097075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:53.097115 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:53.097197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:53.097661 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.098283 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.099119 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.099348 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.099452 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.101986 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59687: BlockingRecv error: recv got EOF from 127.0.98.1:59687 (error 108)
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":98,"threads_started":1}
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:53.288528 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:53.283031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:44809)
0924 07:11:53.283283 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:53.283297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:53.283425 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:53.283675 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:53.283677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:53.283845 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:53.283958 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.283964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.284918 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.284922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.285589 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.285601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.286005 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.286008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.286085 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.286607 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:53.286618 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:53.288363 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
W20260924 07:11:53.288650 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:53.283148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44809 (local address 127.0.98.62:40397)
0924 07:11:53.283317 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:53.283320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:53.283351 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:11:53.283531 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:53.283534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:53.283579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:53.283647 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:53.284082 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.284825 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.285687 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.285901 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.285999 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.288531 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44809: BlockingRecv error: recv got EOF from 127.0.98.2:44809 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:53 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:53.490629 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:53.485469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:45329)
0924 07:11:53.485756 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:53.485771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:53.485872 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:53.486105 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:53.486107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:53.486274 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:53.486381 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.486386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.487080 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.487082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.487691 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:53.487700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.488157 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.488159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:53.488230 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.488725 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:53.488739 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:53.490457 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":132,"threads_started":1}
W20260924 07:11:53.490764 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:53.485560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45329 (local address 127.0.98.62:40397)
0924 07:11:53.485694 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:53.485698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:53.485843 (+ 145us) server_negotiation.cc:407] Connection header received
0924 07:11:53.485961 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:53.485964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:53.486011 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:53.486074 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:53.486467 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.487000 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.487797 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:53.488061 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:53.488142 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:53.490634 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45329: BlockingRecv error: recv got EOF from 127.0.98.3:45329 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:54.107941 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:54.102691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:42699)
0924 07:11:54.102945 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:54.102958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:54.103086 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:54.103310 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:54.103312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:54.103514 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:54.103612 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.103617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.104387 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.104390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.104980 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.104989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.105423 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.105425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.105484 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.105978 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:54.105990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:54.107801 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
W20260924 07:11:54.108068 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:54.102810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42699 (local address 127.0.98.62:40397)
0924 07:11:54.103019 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:54.103023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:54.103043 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:11:54.103158 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:54.103160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:54.103200 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:54.103288 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:54.103726 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.104275 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.105111 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.105335 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.105411 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.107946 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42699: BlockingRecv error: recv got EOF from 127.0.98.1:42699 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:54.294492 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:54.289255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:34681)
0924 07:11:54.289501 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:54.289513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:54.289612 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:54.289811 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:54.289812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:54.289980 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:54.290078 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.290083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.290849 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.290852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.291512 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.291522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.291991 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.291995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.292065 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.292552 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:54.292564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:54.294340 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":71,"threads_started":1}
W20260924 07:11:54.294660 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:54.289317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34681 (local address 127.0.98.62:40397)
0924 07:11:54.289444 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:54.289448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:54.289573 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:11:54.289691 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:54.289694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:54.289736 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:54.289795 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:54.290159 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.290743 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.291657 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.291886 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.291968 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.294525 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34681: BlockingRecv error: recv got EOF from 127.0.98.2:34681 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:54 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:54.497112 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:54.491426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:50723)
0924 07:11:54.491716 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:54.491729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:54.491846 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:54.492132 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:54.492135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:54.492316 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:54.492416 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.492422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.493199 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.493202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.493772 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:54.493781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.494310 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.494314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:54.494380 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.494921 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:54.494933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:54.496962 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":97,"threads_started":1}
W20260924 07:11:54.497256 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:54.491545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50723 (local address 127.0.98.62:40397)
0924 07:11:54.491684 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:54.491688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:54.491815 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:11:54.491919 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:54.491921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:54.491965 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:54.492034 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:54.492537 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.493059 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.493918 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:54.494190 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:54.494280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:54.497145 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50723: BlockingRecv error: recv got EOF from 127.0.98.3:50723 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:55.114477 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:55.108764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:39411)
0924 07:11:55.109101 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:55.109115 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:55.109211 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:55.109513 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:55.109515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:55.109688 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:55.109811 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.109817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.110707 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.110710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.111435 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.111446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.111882 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.111885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.111943 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.112469 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:55.112480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:55.114337 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":81,"threads_started":1}
W20260924 07:11:55.114653 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:55.108889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39411 (local address 127.0.98.62:40397)
0924 07:11:55.109119 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:55.109122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:55.109168 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:11:55.109353 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:55.109357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:55.109404 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:55.109483 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:55.109967 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.110591 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.111556 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.111792 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.111873 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.114499 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39411: BlockingRecv error: recv got EOF from 127.0.98.1:39411 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:55.301121 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:55.295268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:35751)
0924 07:11:55.295627 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:55.295642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:55.295781 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:55.296228 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:55.296231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:55.296403 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:55.296508 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.296513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.297420 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.297424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.297992 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.298002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.298500 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.298502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.298565 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.299125 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:55.299137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:55.300989 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
W20260924 07:11:55.301407 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:55.295337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35751 (local address 127.0.98.62:40397)
0924 07:11:55.295502 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:55.295507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:55.295755 (+ 248us) server_negotiation.cc:407] Connection header received
0924 07:11:55.295881 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:55.295884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:55.295946 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:55.296154 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:55.296645 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.297298 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.298168 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.298399 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.298482 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.301204 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35751: BlockingRecv error: recv got EOF from 127.0.98.2:35751 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:55 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:55.506493 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:55.497929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:50165)
0924 07:11:55.498244 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:55.498259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:55.498369 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:55.498673 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:55.498675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:55.498844 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:55.498943 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.498948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.499816 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.499819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.500417 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:55.500427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.500892 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.500895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:55.500965 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.504505 (+ 3540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:55.504518 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:55.506368 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
W20260924 07:11:55.506656 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:55.498044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50165 (local address 127.0.98.62:40397)
0924 07:11:55.498271 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:55.498275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:55.498317 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:11:55.498486 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:55.498489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:55.498540 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:55.498632 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:55.499078 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.499700 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.500532 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:55.500747 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:55.500824 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:55.506527 (+ 5703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50165: BlockingRecv error: recv got EOF from 127.0.98.3:50165 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:56.120900 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:56.115309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:55699)
0924 07:11:56.115668 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:56.115684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:56.115785 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:56.116015 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:56.116017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:56.116208 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:56.116308 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.116314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.117064 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.117066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.117663 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.117672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.118210 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.118214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.118300 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.118952 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:56.118963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:56.120780 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":152,"threads_started":1}
W20260924 07:11:56.121093 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:56.115445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55699 (local address 127.0.98.62:40397)
0924 07:11:56.115730 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:56.115734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:56.115758 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:11:56.115869 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:56.115872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:56.115914 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:56.115993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:56.116410 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.116975 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.117824 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.118093 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.118195 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.120958 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55699: BlockingRecv error: recv got EOF from 127.0.98.1:55699 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":132,"threads_started":1}
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:56.307273 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:56.302048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:46749)
0924 07:11:56.302319 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:56.302332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:56.302456 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:56.302689 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:56.302691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:56.302857 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:56.302954 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.302959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.303753 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.303756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.304312 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.304321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.304724 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.304727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.304782 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.305233 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:56.305244 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:56.307096 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260924 07:11:56.307323 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:56.302176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46749 (local address 127.0.98.62:40397)
0924 07:11:56.302373 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:56.302376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:56.302392 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:11:56.302537 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:56.302539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:56.302588 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:56.302665 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:56.303038 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.303658 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.304428 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.304643 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.304713 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.307227 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46749: BlockingRecv error: recv got EOF from 127.0.98.2:46749 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:56 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:56.512559 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:56.507213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:39721)
0924 07:11:56.507548 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:56.507564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:56.507670 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:56.507929 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:56.507933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:56.508104 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:56.508201 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.508207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.508999 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.509002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.509569 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:56.509577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.510025 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.510028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:56.510094 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.510604 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:56.510618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:56.512442 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":95,"threads_started":1}
W20260924 07:11:56.512633 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:56.507281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39721 (local address 127.0.98.62:40397)
0924 07:11:56.507438 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:56.507442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:56.507638 (+ 196us) server_negotiation.cc:407] Connection header received
0924 07:11:56.507749 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:56.507752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:56.507806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:56.507891 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:56.508337 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.508871 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.509687 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:56.509923 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:56.510011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:56.512540 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39721: BlockingRecv error: recv got EOF from 127.0.98.3:39721 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:57.127681 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:57.121724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:45323)
0924 07:11:57.122033 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:57.122048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:57.122138 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:57.122466 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:57.122468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:57.122743 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:57.122856 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.122863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.123860 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.123865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.124528 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.124539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.125035 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.125038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.125096 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.125587 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:57.125598 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:57.127540 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
W20260924 07:11:57.127810 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:57.121834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45323 (local address 127.0.98.62:40397)
0924 07:11:57.122115 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:57.122119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:57.122135 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:11:57.122281 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:57.122293 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:57.122343 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:57.122439 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:57.122995 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.123745 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.124658 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.124919 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.125006 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.127694 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45323: BlockingRecv error: recv got EOF from 127.0.98.1:45323 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:57.313685 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:57.308193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:58523)
0924 07:11:57.308477 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:57.308490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:57.308576 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:57.308872 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:57.308875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:57.309050 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:57.309149 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.309154 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.309898 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.309902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.310623 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.310634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.311124 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.311126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.311200 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.311671 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:57.311682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:57.313562 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
W20260924 07:11:57.313822 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:57.308399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58523 (local address 127.0.98.62:40397)
0924 07:11:57.308542 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:57.308546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:57.308560 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:11:57.308668 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:57.308672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:57.308713 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:57.308783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:57.309253 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.309788 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.310732 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.311026 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.311107 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.313708 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58523: BlockingRecv error: recv got EOF from 127.0.98.2:58523 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:57 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:57.518784 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:57.513316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:60809)
0924 07:11:57.513612 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:57.513626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:57.513739 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:57.514009 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:57.514012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:57.514197 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:57.514286 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.514292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.515058 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.515061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.515677 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:57.515686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.516108 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.516112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:57.516191 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.516837 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:57.516866 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:57.518650 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
W20260924 07:11:57.518916 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:57.513409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60809 (local address 127.0.98.62:40397)
0924 07:11:57.513568 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:57.513572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:57.513699 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:11:57.513830 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:57.513833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:57.513881 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:57.513948 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:57.514399 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.514950 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.515788 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:57.516008 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:57.516115 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:57.518799 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60809: BlockingRecv error: recv got EOF from 127.0.98.3:60809 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:58.133972 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:58.128425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:58185)
0924 07:11:58.128785 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:58.128799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:58.128909 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:58.129143 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:58.129145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:58.129317 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:58.129417 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.129423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.130240 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.130243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.130888 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.130898 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.131481 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.131485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.131586 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.132065 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:58.132077 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:58.133853 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":172,"threads_started":1}
W20260924 07:11:58.134160 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:58.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58185 (local address 127.0.98.62:40397)
0924 07:11:58.128805 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:58.128809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:58.128872 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:11:58.128992 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:58.128994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:58.129039 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:58.129115 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:58.129544 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.130138 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.131008 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.131268 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.131348 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.134018 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58185: BlockingRecv error: recv got EOF from 127.0.98.1:58185 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:58.320639 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:58.314514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:40221)
0924 07:11:58.314793 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:58.314810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:58.314929 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:58.315256 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:58.315259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:58.315550 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:58.315672 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.315680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.316669 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.316672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.317388 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.317398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.317838 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.317840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.317897 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.318388 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:58.318402 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:58.320508 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
W20260924 07:11:58.320765 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:58.314706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40221 (local address 127.0.98.62:40397)
0924 07:11:58.314875 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:58.314880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:58.314895 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:11:58.315034 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:58.315038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:58.315092 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:58.315172 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:58.315798 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.316557 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.317517 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.317747 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.317833 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.320659 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40221: BlockingRecv error: recv got EOF from 127.0.98.2:40221 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:58 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:58.524492 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:58.519463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:36357)
0924 07:11:58.519731 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:58.519743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:58.519850 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:58.520087 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:58.520090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:58.520274 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:58.520363 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.520368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.521135 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.521139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.521698 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:58.521706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.522131 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.522135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:58.522212 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.522670 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:58.522681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:58.524345 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260924 07:11:58.524581 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:58.519533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36357 (local address 127.0.98.62:40397)
0924 07:11:58.519674 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:58.519678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:58.519807 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:11:58.519934 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:58.519937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:58.519986 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:58.520057 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:58.520469 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.521033 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.521803 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.522037 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.522120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.524479 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36357: BlockingRecv error: recv got EOF from 127.0.98.3:36357 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:59.140209 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:59.134692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:43515)
0924 07:11:59.134990 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:59.135003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:59.135097 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:59.135338 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:59.135340 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:59.135549 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:59.135654 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.135661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.136408 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.136411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.137052 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.137063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.137459 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.137461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.137518 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.138140 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:59.138152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:59.140038 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
W20260924 07:11:59.140319 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:59.134819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43515 (local address 127.0.98.62:40397)
0924 07:11:59.135047 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:59.135051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:59.135065 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:11:59.135179 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:59.135181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:59.135222 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:59.135315 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:59.135760 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.136297 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.137160 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.137377 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.137451 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.140194 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43515: BlockingRecv error: recv got EOF from 127.0.98.1:43515 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:59.326954 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:59.321430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:36995)
0924 07:11:59.321811 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:59.321824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:59.321928 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:59.322147 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:59.322149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:59.322323 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:59.322420 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.322433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.323202 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.323205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.323855 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.323864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.324317 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.324320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.324384 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.324908 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:59.324920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:59.326837 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":155,"threads_started":1}
W20260924 07:11:59.327086 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:59.321500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36995 (local address 127.0.98.62:40397)
0924 07:11:59.321634 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:59.321639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:59.321899 (+ 260us) server_negotiation.cc:407] Connection header received
0924 07:11:59.322000 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:59.322002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:59.322045 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:59.322115 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:59.322519 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.323088 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.323962 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.324211 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.324295 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.326968 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36995: BlockingRecv error: recv got EOF from 127.0.98.2:36995 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:11:59 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:11:59.530947 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:59.525478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:55295)
0924 07:11:59.525879 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:59.525893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:59.526009 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:59.526250 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:59.526252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:59.526439 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:59.526552 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.526558 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.527442 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.527445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.528009 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.528017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.528437 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.528440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.528520 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.528949 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:59.528959 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:59.530830 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":133,"threads_started":1}
W20260924 07:11:59.531052 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:11:59.525608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55295 (local address 127.0.98.62:40397)
0924 07:11:59.525768 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:59.525772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:59.525971 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:11:59.526100 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:59.526103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:59.526150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:59.526227 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:59.526665 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.527313 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.528114 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.528343 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.528426 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.530953 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55295: BlockingRecv error: recv got EOF from 127.0.98.3:55295 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:00.146656 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.141008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:60007)
0924 07:12:00.141335 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.141349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.141446 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.141763 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.141765 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.141939 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.142037 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.142043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.142865 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.142868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.143609 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.143623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.144196 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.144199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.144279 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.144786 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.144798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.146531 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":120,"threads_started":1}
W20260924 07:12:00.146847 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:00.141191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60007 (local address 127.0.98.62:40397)
0924 07:12:00.141421 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.141424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.141438 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:00.141581 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.141584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.141635 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.141729 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.142178 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.142732 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.143732 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.144038 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.144144 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.146714 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60007: BlockingRecv error: recv got EOF from 127.0.98.1:60007 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":59,"threads_started":1}
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:00.333469 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.327729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:58999)
0924 07:12:00.328021 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.328036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.328144 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.328377 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.328379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.328561 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.328654 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.328659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.329548 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.329551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.330125 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.330135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.330587 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.330590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.330654 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.331199 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.331214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.333354 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":69,"threads_started":1}
W20260924 07:12:00.333617 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:00.327798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58999 (local address 127.0.98.62:40397)
0924 07:12:00.327992 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.327997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.328101 (+ 104us) server_negotiation.cc:407] Connection header received
0924 07:12:00.328227 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.328230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.328275 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.328338 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.328761 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.329440 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.330222 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.330498 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.330584 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.333507 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58999: BlockingRecv error: recv got EOF from 127.0.98.2:58999 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:00 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:00.541797 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:55961)
0924 07:12:00.532159 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.532173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.532280 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.532506 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.532508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.532676 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.532774 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.532779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.533537 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.533541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.534176 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.534186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.534652 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.534656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.534751 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.539429 (+ 4678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.539445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.541669 (+ 2224us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260924 07:12:00.542086 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:00.531938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55961 (local address 127.0.98.62:40397)
0924 07:12:00.532084 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.532097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.532246 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:12:00.532364 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.532367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.532419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.532479 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.532886 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.533413 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.534308 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.534543 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.534629 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.541967 (+ 7338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55961: BlockingRecv error: recv got EOF from 127.0.98.3:55961 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:01.153534 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.147574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:43315)
0924 07:12:01.148097 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:01.148113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:01.148227 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:01.148600 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:01.148603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:01.148792 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:01.148891 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.148899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.149775 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.149778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.150360 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.150368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.150847 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.150850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.150921 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.151401 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:01.151414 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:01.153408 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":87,"threads_started":1}
W20260924 07:12:01.153688 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:01.147721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43315 (local address 127.0.98.62:40397)
0924 07:12:01.148113 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:01.148118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:01.148172 (+ 54us) server_negotiation.cc:407] Connection header received
0924 07:12:01.148359 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:01.148362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:01.148417 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:01.148551 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:01.149015 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.149653 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.150491 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.150726 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.150883 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.153554 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43315: BlockingRecv error: recv got EOF from 127.0.98.1:43315 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":86,"threads_started":1}
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:01.339828 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.334248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:38159)
0924 07:12:01.334528 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:01.334541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:01.334644 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:01.334976 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:01.334979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:01.335154 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:01.335268 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.335273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.336024 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.336027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.336593 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.336604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.337127 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.337131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.337214 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.337751 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:01.337766 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:01.339681 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
W20260924 07:12:01.339907 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:01.334339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38159 (local address 127.0.98.62:40397)
0924 07:12:01.334467 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:01.334471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:01.334612 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:12:01.334727 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:01.334729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:01.334777 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:01.334855 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:01.335397 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.335917 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.336705 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.337017 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.337111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.339810 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38159: BlockingRecv error: recv got EOF from 127.0.98.2:38159 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:01 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:01.548110 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.542651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:58011)
0924 07:12:01.542958 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:01.542973 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:01.543066 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:01.543412 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:01.543415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:01.543618 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:01.543725 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.543732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.544480 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.544483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.545100 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.545109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.545572 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.545574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.545631 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.546162 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:01.546174 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:01.547972 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":147,"threads_started":1}
W20260924 07:12:01.548403 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:01.542823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58011 (local address 127.0.98.62:40397)
0924 07:12:01.543016 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:01.543019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:01.543034 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:12:01.543144 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:01.543147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:01.543190 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:01.543267 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:01.543841 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.544363 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.545233 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.545458 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.545545 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.548190 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58011: BlockingRecv error: recv got EOF from 127.0.98.3:58011 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:02.159726 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.154278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:38495)
0924 07:12:02.154594 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.154608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.154712 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.154929 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.154931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.155119 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:02.155219 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.155225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.156067 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.156069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.156637 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.156645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.157116 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.157119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.157187 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.157777 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:02.157790 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:02.159606 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":72,"threads_started":1}
W20260924 07:12:02.159901 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:02.154351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38495 (local address 127.0.98.62:40397)
0924 07:12:02.154593 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.154596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.154677 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:12:02.154785 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.154787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.154825 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.154899 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:02.155308 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.155885 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.156769 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.157004 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.157088 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.159767 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38495: BlockingRecv error: recv got EOF from 127.0.98.1:38495 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:02.345527 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.340666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:46361)
0924 07:12:02.340924 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.340938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.341028 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.341240 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.341242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.341412 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:02.341506 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.341511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.342221 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.342223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.342771 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.342778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.343142 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.343145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.343198 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.343641 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:02.343652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:02.345404 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
W20260924 07:12:02.345659 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:02.340744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46361 (local address 127.0.98.62:40397)
0924 07:12:02.340867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.340870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.340986 (+ 116us) server_negotiation.cc:407] Connection header received
0924 07:12:02.341111 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.341113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.341154 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.341209 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:02.341584 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.342097 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.342854 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.343062 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.343156 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.345541 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46361: BlockingRecv error: recv got EOF from 127.0.98.2:46361 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:02 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:02.554286 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.548944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:32967)
0924 07:12:02.549257 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.549272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.549373 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.549632 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.549635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.549850 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:02.549943 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.549949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.550688 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.550691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.551258 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.551266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.551727 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.551730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.551789 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.552242 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:02.552253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:02.554133 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
W20260924 07:12:02.554386 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:02.549154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32967 (local address 127.0.98.62:40397)
0924 07:12:02.549270 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.549274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.549344 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:12:02.549453 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.549457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.549507 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.549569 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:02.550052 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.550572 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.551353 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.551615 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.551722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.554282 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32967: BlockingRecv error: recv got EOF from 127.0.98.3:32967 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:03.165982 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:03.160647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:52007)
0924 07:12:03.160935 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:03.160950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:03.161046 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:03.161321 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:03.161324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:03.161527 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:03.161644 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.161651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.162471 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.162475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.163028 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.163036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.163518 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.163521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.163602 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.164086 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:03.164098 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:03.165809 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
W20260924 07:12:03.166085 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:03.160776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52007 (local address 127.0.98.62:40397)
0924 07:12:03.161029 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:03.161033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:03.161047 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:03.161146 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:03.161148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:03.161198 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:03.161302 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:03.161768 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.162365 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.163139 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.163419 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.163510 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.165952 (+ 2442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52007: BlockingRecv error: recv got EOF from 127.0.98.1:52007 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:03.352170 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:03.346374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:33111)
0924 07:12:03.346743 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:03.346755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:03.346863 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:03.347237 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:03.347240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:03.347483 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:03.347592 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.347598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.348453 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.348456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.349029 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.349038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.349522 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.349525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.349594 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.350187 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:03.350201 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:03.352047 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":79,"threads_started":1}
W20260924 07:12:03.352341 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:03.346487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33111 (local address 127.0.98.62:40397)
0924 07:12:03.346630 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:03.346634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:03.346837 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:12:03.346949 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:03.346951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:03.347001 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:03.347078 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:03.347742 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.348315 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.349168 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.349412 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.349494 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.352217 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33111: BlockingRecv error: recv got EOF from 127.0.98.2:33111 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:03 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:03.560570 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:03.555242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:55793)
0924 07:12:03.555506 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:03.555521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:03.555621 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:03.555927 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:03.555930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:03.556101 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:03.556198 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.556205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.557008 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.557011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.557576 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:03.557585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.558040 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.558043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:03.558099 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.558553 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:03.558564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:03.560448 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260924 07:12:03.560770 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:03.555272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55793 (local address 127.0.98.62:40397)
0924 07:12:03.555466 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:03.555470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:03.555575 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:12:03.555735 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:03.555737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:03.555782 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:03.555857 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:03.556343 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.556903 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.557695 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:03.557946 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:03.558027 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:03.560597 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55793: BlockingRecv error: recv got EOF from 127.0.98.3:55793 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:04.172113 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.166782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:48707)
0924 07:12:04.167039 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.167055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.167163 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.167476 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.167479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.167719 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.167842 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.167851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.168575 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.168577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.169162 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.169172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.169649 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.169652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.169739 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.170300 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.170313 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.171997 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
W20260924 07:12:04.172322 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:04.166872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48707 (local address 127.0.98.62:40397)
0924 07:12:04.167100 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.167104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.167120 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:12:04.167261 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.167264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.167328 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.167451 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.167956 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.168485 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.169251 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.169527 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.169608 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.172134 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48707: BlockingRecv error: recv got EOF from 127.0.98.1:48707 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:04.358610 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.353009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:48855)
0924 07:12:04.353319 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.353332 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.353429 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.353696 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.353698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.353867 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.353970 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.353975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.354790 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.354794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.355513 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.355525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.355964 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.355967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.356028 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.356548 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.356560 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.358469 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
W20260924 07:12:04.358723 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:04.353125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48855 (local address 127.0.98.62:40397)
0924 07:12:04.353324 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.353328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.353396 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:12:04.353544 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.353547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.353597 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.353673 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.354080 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.354660 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.355632 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.355858 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.355941 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.358609 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48855: BlockingRecv error: recv got EOF from 127.0.98.2:48855 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:04 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:04.566488 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.561387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:52235)
0924 07:12:04.561647 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.561659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.561757 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.561993 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.561995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.562167 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.562261 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.562267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.563019 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.563022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.563646 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.563655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.564109 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.564112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.564175 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.564685 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.564696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.566367 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260924 07:12:04.566634 9611 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:04.561499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52235 (local address 127.0.98.62:40397)
0924 07:12:04.561651 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.561654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.561726 (+ 72us) server_negotiation.cc:407] Connection header received
0924 07:12:04.561828 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.561830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.561874 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.561959 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.562382 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.562920 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.563768 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.564008 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.564094 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.566512 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52235: BlockingRecv error: recv got EOF from 127.0.98.3:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:05.177790 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:05.172800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:41457)
0924 07:12:05.173066 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:05.173079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:05.173161 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:05.173420 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:05.173423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:05.173592 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:05.173686 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.173691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.174421 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.174424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.174984 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.174992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.175423 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.175426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.175488 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.175982 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:05.175993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:05.177674 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
W20260924 07:12:05.177917 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:05.172872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41457 (local address 127.0.98.62:40397)
0924 07:12:05.173098 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:05.173101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:05.173115 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:05.173234 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:05.173236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:05.173278 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:05.173353 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:05.173783 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.174303 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.175079 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.175292 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.175361 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.177805 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41457: BlockingRecv error: recv got EOF from 127.0.98.1:41457 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:05.364799 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:05.359593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:46653)
0924 07:12:05.359866 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:05.359882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:05.360001 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:05.360238 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:05.360240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:05.360435 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:05.360533 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.360539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.361279 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.361283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.361851 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.361860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.362278 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.362282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.362358 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.362880 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:05.362891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:05.364676 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
W20260924 07:12:05.364938 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:05.359760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46653 (local address 127.0.98.62:40397)
0924 07:12:05.359882 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:05.359886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:05.359937 (+ 51us) server_negotiation.cc:407] Connection header received
0924 07:12:05.360087 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:05.360090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:05.360133 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:05.360194 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:05.360647 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.361169 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.361966 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.362179 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.362262 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.364819 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46653: BlockingRecv error: recv got EOF from 127.0.98.2:46653 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:05 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:05.576431 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:05.567434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:55149)
0924 07:12:05.567748 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:05.567765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:05.567902 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:05.568183 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:05.568185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:05.568353 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:05.568516 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.568523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.569309 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.569312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.570032 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:05.570041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.570574 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.570577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:05.570642 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.574373 (+ 3731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:05.574388 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:05.576308 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
W20260924 07:12:05.576633 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:05.567631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55149 (local address 127.0.98.62:40397)
0924 07:12:05.567790 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:05.567795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:05.567845 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:12:05.568007 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:05.568012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:05.568061 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:05.568133 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:05.568648 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.569186 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.570165 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:05.570455 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:05.570563 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:05.576512 (+ 5949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55149: BlockingRecv error: recv got EOF from 127.0.98.3:55149 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:06.183874 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:06.178594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:52743)
0924 07:12:06.178879 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:06.178892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:06.178989 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:06.179266 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:06.179267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:06.179482 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:06.179584 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.179591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.180312 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.180315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.180906 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.180915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.181396 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.181399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.181467 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.181913 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:06.181924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:06.183722 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":105,"threads_started":1}
W20260924 07:12:06.183977 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:06.178640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52743 (local address 127.0.98.62:40397)
0924 07:12:06.178927 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:06.178931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:06.178944 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:12:06.179109 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:06.179112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:06.179161 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:06.179244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:06.179684 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.180202 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.181043 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.181270 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.181349 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.183850 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52743: BlockingRecv error: recv got EOF from 127.0.98.1:52743 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":58,"threads_started":1}
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:06.370862 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:06.365603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:45809)
0924 07:12:06.365880 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:06.365892 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:06.366002 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:06.366242 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:06.366244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:06.366441 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:06.366585 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.366592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.367314 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.367318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.367938 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.367947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.368377 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.368380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.368456 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.368987 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:06.368998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:06.370726 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":99,"threads_started":1}
W20260924 07:12:06.370972 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:06.365646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45809 (local address 127.0.98.62:40397)
0924 07:12:06.365793 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:06.365798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:06.365958 (+ 160us) server_negotiation.cc:407] Connection header received
0924 07:12:06.366093 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:06.366096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:06.366142 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:06.366206 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:06.366700 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.367215 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.368052 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.368279 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.368363 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.370871 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45809: BlockingRecv error: recv got EOF from 127.0.98.2:45809 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:06 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:06.582368 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:06.577228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.3:42631)
0924 07:12:06.577524 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:06.577541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:06.577649 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:06.577893 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:06.577896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:06.578099 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:06.578189 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.578195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.578921 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.578925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.579530 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:06.579539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.579943 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.579947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:06.580036 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.580549 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:06.580561 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:06.582249 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":99,"threads_started":1}
W20260924 07:12:06.582508 9619 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:06.577360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42631 (local address 127.0.98.62:40397)
0924 07:12:06.577598 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:06.577602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:06.577617 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:12:06.577744 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:06.577747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:06.577794 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:06.577857 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:06.578310 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.578825 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.579633 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:06.579847 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:06.579932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:06.582388 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42631: BlockingRecv error: recv got EOF from 127.0.98.3:42631 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 07:12:07 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:07 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:07.190362 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.184795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.1:51203)
0924 07:12:07.185058 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:07.185074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:07.185164 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:07.185431 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:07.185433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:07.185661 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:07.185777 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.185784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.186563 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.186566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.187140 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.187148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.187586 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.187589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.187660 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.188159 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:07.188171 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:07.190185 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":82,"threads_started":1}
W20260924 07:12:07.190451 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:07.184912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51203 (local address 127.0.98.62:40397)
0924 07:12:07.185123 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:07.185127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:07.185149 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:12:07.185244 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:07.185247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:07.185286 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:07.185367 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:07.185887 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.186439 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.187244 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.187488 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.187564 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.190334 (+ 2770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51203: BlockingRecv error: recv got EOF from 127.0.98.1:51203 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":56,"threads_started":1}
Sep 24 07:12:07 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:12:07 dist-test-slave-cnv3 krb5kdc[8565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:12:07.377086 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.371852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.98.2:39205)
0924 07:12:07.372129 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:07.372143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:07.372259 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:07.372466 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:07.372468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:07.372644 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:07.372741 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.372746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.373471 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.373474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.374057 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.374066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.374502 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.374504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.374564 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.375001 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:07.375012 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:07.376947 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40397: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
W20260924 07:12:07.377218 9623 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:07.371983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39205 (local address 127.0.98.62:40397)
0924 07:12:07.372133 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:07.372137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:07.372195 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:12:07.372333 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:07.372335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:07.372374 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:07.372439 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:07.372834 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.373368 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.374158 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.374414 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.374494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.377099 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39205: BlockingRecv error: recv got EOF from 127.0.98.2:39205 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:12:07.474157 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9148
I20260924 07:12:07.479149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9300
I20260924 07:12:07.483596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9433
I20260924 07:12:07.488046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9078
2026-09-24T07:12:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17882 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T07:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:07Z Disabled control of system clock
I20260924 07:12:07.518715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40985
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:12:07.633882 9628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:07.634155 9628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:07.634187 9628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:07.638315 9628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:07.638384 9628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:07.638399 9628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:07.638414 9628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:07.638428 9628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:07.643240 9628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:07.644359 9628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:07.645779 9628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:07.654528 9633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:07.654528 9634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:07.654584 9628 server_base.cc:1034] running on GCE node
W20260924 07:12:07.654529 9636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:07.655274 9628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:07.656486 9628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:07.657668 9628 hybrid_clock.cc:648] HybridClock initialized: now 1790233927657630 us; error 52 us; skew 500 ppm
I20260924 07:12:07.659741 9628 webserver.cc:460] Webserver started at http://127.0.98.62:37413/ using document root <none> and password file <none>
I20260924 07:12:07.660307 9628 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:07.660375 9628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:07.660580 9628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:07.662711 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "93bb2853719041fbbfa16341aea0cefc"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "3205567a73144c4cbb2e4e7b917ce34c"
server_key_iv: "26d14fd9f8c13e3619d2f8afb94c14eb"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.663259 9628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "93bb2853719041fbbfa16341aea0cefc"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "3205567a73144c4cbb2e4e7b917ce34c"
server_key_iv: "26d14fd9f8c13e3619d2f8afb94c14eb"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.667068 9628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:12:07.669744 9642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.671063 9628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:12:07.671217 9628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "93bb2853719041fbbfa16341aea0cefc"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "3205567a73144c4cbb2e4e7b917ce34c"
server_key_iv: "26d14fd9f8c13e3619d2f8afb94c14eb"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.671325 9628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:07.687219 9628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:07.687978 9628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:07.688145 9628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:07.695003 9628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40985
I20260924 07:12:07.695024 9694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40985 every 8 connection(s)
I20260924 07:12:07.696002 9628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:07.697671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9628
I20260924 07:12:07.697814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:07.698061 392 external_mini_cluster.cc:1428] Setting key 182f7c50593e666691046451bb56c966
I20260924 07:12:07.699576 9695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:07.706626 9695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc: Bootstrap starting.
I20260924 07:12:07.708855 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.699447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.98.62:40985)
0924 07:12:07.700262 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:07.700283 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:07.700326 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:12:07.701162 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:07.701185 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:07.701715 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:07.702101 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:07.702562 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.704008 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.704931 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.705652 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.705814 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.706703 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:07.706710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:07.706728 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:07.707235 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:07.707308 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:07.707334 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:07.707448 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:07.707604 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:12:07.707886 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":226,"threads_started":1}
I20260924 07:12:07.709378 9695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:07.710142 9695 log.cc:824] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:07.712162 9695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc: No bootstrap required, opened a new log
I20260924 07:12:07.715795 9695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } }
I20260924 07:12:07.716051 9695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:07.716112 9695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93bb2853719041fbbfa16341aea0cefc, State: Initialized, Role: FOLLOWER
I20260924 07:12:07.716570 9695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } }
I20260924 07:12:07.716719 9695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:07.716785 9695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:07.716881 9695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:07.717806 9695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } }
I20260924 07:12:07.718149 9695 leader_election.cc:302] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93bb2853719041fbbfa16341aea0cefc; no voters:
I20260924 07:12:07.718457 9695 leader_election.cc:288] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:07.718637 9700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:07.718928 9700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [term 1 LEADER]: Becoming Leader. State: Replica: 93bb2853719041fbbfa16341aea0cefc, State: Running, Role: LEADER
I20260924 07:12:07.719236 9700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } }
I20260924 07:12:07.719733 9695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:07.720820 9702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93bb2853719041fbbfa16341aea0cefc. Latest consensus state: current_term: 1 leader_uuid: "93bb2853719041fbbfa16341aea0cefc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } } }
I20260924 07:12:07.720952 9702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:07.721225 9701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93bb2853719041fbbfa16341aea0cefc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93bb2853719041fbbfa16341aea0cefc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40985 } } }
I20260924 07:12:07.721319 9701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:07.721398 9709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:07.724614 9709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:07.730648 9709 catalog_manager.cc:1347] Generated new cluster ID: 2dff30fc93754a2d8da1ceccebee761d
I20260924 07:12:07.730811 9709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:07.746824 9709 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:07.747915 9709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:07.755064 9709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93bb2853719041fbbfa16341aea0cefc: Generated new TSK 0
I20260924 07:12:07.756048 9709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:12:07.759473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:12:07.875968 9719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:07.876255 9719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:07.876307 9719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:07.880447 9719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:07.880512 9719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:07.880587 9719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:07.885407 9719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:07.886497 9719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:07.887967 9719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:07.897264 9727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:07.897272 9725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:07.897281 9724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:07.897486 9719 server_base.cc:1034] running on GCE node
I20260924 07:12:07.897929 9719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:07.898551 9719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:07.899703 9719 hybrid_clock.cc:648] HybridClock initialized: now 1790233927899683 us; error 38 us; skew 500 ppm
I20260924 07:12:07.901523 9719 webserver.cc:460] Webserver started at http://127.0.98.1:44871/ using document root <none> and password file <none>
I20260924 07:12:07.902074 9719 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:07.902141 9719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:07.902320 9719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:07.904369 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "ca79ddedba0e4275b51e41162e20a733"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "876bbb08aac4f7faddf82fb942cc6c10"
server_key_iv: "61681828d09a7e9a7c89ee112532b3ad"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.904908 9719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ca79ddedba0e4275b51e41162e20a733"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "876bbb08aac4f7faddf82fb942cc6c10"
server_key_iv: "61681828d09a7e9a7c89ee112532b3ad"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.908641 9719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:12:07.911466 9733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.912948 9719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:07.913074 9719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "ca79ddedba0e4275b51e41162e20a733"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-cnv3"
server_key: "876bbb08aac4f7faddf82fb942cc6c10"
server_key_iv: "61681828d09a7e9a7c89ee112532b3ad"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.913196 9719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:07.940064 9719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:07.940781 9719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:07.940964 9719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:07.941553 9719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:07.942536 9719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:07.942585 9719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.942641 9719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:07.942668 9719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.951320 9719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45209
I20260924 07:12:07.951354 9846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45209 every 8 connection(s)
I20260924 07:12:07.952272 9719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:07.955978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9719
I20260924 07:12:07.956089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:07.956375 392 external_mini_cluster.cc:1428] Setting key ad41912280eeddd0f7d2059368e6463a
I20260924 07:12:07.958411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:12:07.962504 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.954187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38913 (local address 127.0.98.62:40985)
0924 07:12:07.954326 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:07.954330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:07.955449 (+ 1119us) server_negotiation.cc:407] Connection header received
0924 07:12:07.956986 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:07.956989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:07.957042 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:07.957131 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:07.958146 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.958887 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.960406 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.960691 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.960800 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.962030 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:07.962034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:07.962036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:07.962049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:07.962069 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:07.962077 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:07.962139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:07.962318 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 07:12:07.962389 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:12:07.963727 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.954609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40985 (local address 127.0.98.1:38913)
0924 07:12:07.955294 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:07.955338 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:07.956760 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:07.957332 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:07.957339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:07.957427 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:07.958009 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.958024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.959009 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.959012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.960234 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:07.960279 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.960857 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.960861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:07.961047 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.961715 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:07.961717 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:07.961775 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:07.961859 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:07.962157 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:07.962165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:07.962172 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:12:07.962313 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 07:12:07.962630 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":111,"threads_started":1}
I20260924 07:12:07.965016 9847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40985
I20260924 07:12:07.965394 9847 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:07.965922 9847 heartbeater.cc:511] Master 127.0.98.62:40985 requested a full tablet report, sending...
I20260924 07:12:07.967429 9658 ts_manager.cc:194] Registered new tserver with Master: ca79ddedba0e4275b51e41162e20a733 (127.0.98.1:45209)
I20260924 07:12:07.968551 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38913
W20260924 07:12:08.067672 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.067962 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.068014 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.072157 9850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.072216 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.072290 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:08.077114 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:08.078186 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.079609 9850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.088946 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.088949 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.089003 9850 server_base.cc:1034] running on GCE node
W20260924 07:12:08.088979 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.089534 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.090251 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.091432 9850 hybrid_clock.cc:648] HybridClock initialized: now 1790233928091413 us; error 29 us; skew 500 ppm
I20260924 07:12:08.093334 9850 webserver.cc:460] Webserver started at http://127.0.98.2:40385/ using document root <none> and password file <none>
I20260924 07:12:08.093924 9850 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.093997 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.094205 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.096374 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "820fb843f39f44deb6ffbcfd2849ef0e"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "01254f8cb0970aa46842efc468c93974"
server_key_iv: "d73d2d285f501a067965b7b3f8b499ea"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.096913 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "820fb843f39f44deb6ffbcfd2849ef0e"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "01254f8cb0970aa46842efc468c93974"
server_key_iv: "d73d2d285f501a067965b7b3f8b499ea"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.100596 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:12:08.103206 9864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.104454 9850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:08.104568 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "820fb843f39f44deb6ffbcfd2849ef0e"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "01254f8cb0970aa46842efc468c93974"
server_key_iv: "d73d2d285f501a067965b7b3f8b499ea"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.104674 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.127123 9850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.127888 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.128062 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.128650 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:08.129582 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:08.129628 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.129679 9850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:08.129709 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.138103 9850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33785
I20260924 07:12:08.138142 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33785 every 8 connection(s)
I20260924 07:12:08.139014 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:08.144950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9850
I20260924 07:12:08.145044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:08.145278 392 external_mini_cluster.cc:1428] Setting key 2b0f65a69abd208e4268c5ee42e3135e
I20260924 07:12:08.147266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:12:08.148646 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.140990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48091 (local address 127.0.98.62:40985)
0924 07:12:08.141146 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.141149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.142210 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 07:12:08.143690 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.143693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.143754 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.143827 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.144735 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.145296 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.146547 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.146811 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.146964 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.148182 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.148185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.148187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:08.148198 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.148216 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:08.148224 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.148283 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.148438 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.148502 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:12:08.149562 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.141383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40985 (local address 127.0.98.2:48091)
0924 07:12:08.142051 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:08.142091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:08.143470 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:08.144010 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:08.144017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:08.144105 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:08.144601 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.144614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.145417 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.145420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.146433 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.146444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.147013 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.147018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.147185 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.147921 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:08.147923 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:08.147966 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:08.148035 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:08.148294 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:08.148303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:08.148309 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:12:08.148433 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 07:12:08.148742 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":120,"threads_started":1}
I20260924 07:12:08.150802 9978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40985
I20260924 07:12:08.151080 9978 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:08.151803 9978 heartbeater.cc:511] Master 127.0.98.62:40985 requested a full tablet report, sending...
I20260924 07:12:08.153090 9658 ts_manager.cc:194] Registered new tserver with Master: 820fb843f39f44deb6ffbcfd2849ef0e (127.0.98.2:33785)
I20260924 07:12:08.153882 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48091
W20260924 07:12:08.274847 9981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.275194 9981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.275236 9981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.279520 9981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.279584 9981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.279640 9981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:08.284552 9981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:08.285844 9981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.287268 9981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.297070 9987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.297108 9986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.297165 9981 server_base.cc:1034] running on GCE node
W20260924 07:12:08.297107 9989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.297729 9981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.298334 9981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.299437 9981 hybrid_clock.cc:648] HybridClock initialized: now 1790233928299407 us; error 46 us; skew 500 ppm
I20260924 07:12:08.301285 9981 webserver.cc:460] Webserver started at http://127.0.98.3:46145/ using document root <none> and password file <none>
I20260924 07:12:08.301891 9981 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.301957 9981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.302155 9981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.304241 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "963be2a639ba461aa15d84aef9e658d7"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "0122d908b5b5e60fc1ba853292410675"
server_key_iv: "ec3c31b4bd17f0805fe394ea50afa71a"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.304773 9981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "963be2a639ba461aa15d84aef9e658d7"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "0122d908b5b5e60fc1ba853292410675"
server_key_iv: "ec3c31b4bd17f0805fe394ea50afa71a"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.308468 9981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:12:08.310992 9995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.312363 9981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:08.312543 9981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "963be2a639ba461aa15d84aef9e658d7"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "0122d908b5b5e60fc1ba853292410675"
server_key_iv: "ec3c31b4bd17f0805fe394ea50afa71a"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.312743 9981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.329624 9981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.330353 9981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.330524 9981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.331115 9981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:08.332104 9981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:08.332159 9981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.332211 9981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:08.332238 9981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.340673 9981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40245
I20260924 07:12:08.340735 10108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40245 every 8 connection(s)
I20260924 07:12:08.341624 9981 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:08.343778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 9981
I20260924 07:12:08.343869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:08.344095 392 external_mini_cluster.cc:1428] Setting key 2b08f3229f9fcc25eb90af18b86b2c5f
I20260924 07:12:08.351060 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.343547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39473 (local address 127.0.98.62:40985)
0924 07:12:08.343694 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.343699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.344934 (+ 1235us) server_negotiation.cc:407] Connection header received
0924 07:12:08.346306 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.346308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.346354 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.346419 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.347320 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.347916 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.349148 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.349366 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.349455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.350550 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.350552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.350554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:08.350566 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.350582 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:08.350589 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.350644 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.350883 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.350939 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:12:08.352008 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.344022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40985 (local address 127.0.98.3:39473)
0924 07:12:08.344788 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:08.344843 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:08.346109 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:08.346560 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:08.346567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:08.346653 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:08.347210 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.347225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.348045 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.348051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.349028 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.349038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.349531 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.349534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.349688 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.350304 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:08.350305 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:08.350347 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:08.350418 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:08.350693 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:08.350702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:08.350707 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:08.350890 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 07:12:08.351137 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":141,"threads_started":1}
I20260924 07:12:08.353205 10109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40985
I20260924 07:12:08.353447 10109 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:08.353981 10109 heartbeater.cc:511] Master 127.0.98.62:40985 requested a full tablet report, sending...
I20260924 07:12:08.354975 9658 ts_manager.cc:194] Registered new tserver with Master: 963be2a639ba461aa15d84aef9e658d7 (127.0.98.3:40245)
I20260924 07:12:08.355584 9658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39473
I20260924 07:12:08.357756 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:12:08.360534 9698 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:08.359633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57598 (local address 127.0.98.62:40985)
0924 07:12:08.359729 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.359733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.359999 (+ 266us) server_negotiation.cc:407] Connection header received
0924 07:12:08.360093 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.360095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.360137 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.360223 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.360414 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57598: BlockingRecv error: recv got EOF from 127.0.0.1:57598 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 07:12:08.360678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9719
I20260924 07:12:08.365548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9850
I20260924 07:12:08.370242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9981
I20260924 07:12:08.374987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 9628
2026-09-24T07:12:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (884 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T07:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:08Z Disabled control of system clock
I20260924 07:12:08.401739 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43647
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:12:08.510516 10120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.510788 10120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.510824 10120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.514801 10120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:08.514871 10120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.514885 10120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.514901 10120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:08.514914 10120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:08.519848 10120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:08.520928 10120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.522329 10120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.531104 10126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.531168 10120 server_base.cc:1034] running on GCE node
W20260924 07:12:08.531211 10128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.531105 10125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.531972 10120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.533116 10120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.534271 10120 hybrid_clock.cc:648] HybridClock initialized: now 1790233928534255 us; error 29 us; skew 500 ppm
I20260924 07:12:08.536222 10120 webserver.cc:460] Webserver started at http://127.0.98.62:42909/ using document root <none> and password file <none>
I20260924 07:12:08.536810 10120 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.536877 10120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.537072 10120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.539565 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "a7f8ea67eddb4e8985123424074c76ac"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "71ce278fc768cd45ece21fc85689a525"
server_key_iv: "053068d40cc298c911484c8aabf81c1f"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.540172 10120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7f8ea67eddb4e8985123424074c76ac"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "71ce278fc768cd45ece21fc85689a525"
server_key_iv: "053068d40cc298c911484c8aabf81c1f"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.544153 10120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:12:08.547247 10134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.548832 10120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:08.548982 10120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "a7f8ea67eddb4e8985123424074c76ac"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "71ce278fc768cd45ece21fc85689a525"
server_key_iv: "053068d40cc298c911484c8aabf81c1f"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.549124 10120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.554076 10120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.554736 10120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.554898 10120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.561854 10120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43647
I20260924 07:12:08.561900 10186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43647 every 8 connection(s)
I20260924 07:12:08.562816 10120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:08.566114 10187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:08.567324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10120
I20260924 07:12:08.567476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:08.567771 392 external_mini_cluster.cc:1428] Setting key 5be40da5ed42e76fc6c835e27ca38f0f
I20260924 07:12:08.572691 10187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac: Bootstrap starting.
I20260924 07:12:08.573493 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.569206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.62:43647)
0924 07:12:08.569832 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.569848 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.569884 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:12:08.570557 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.570576 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.571016 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.571325 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.572137 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.572141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.572155 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:08.572165 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.572219 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:08.572236 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.572348 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.572447 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.572697 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260924 07:12:08.575158 10187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:08.576026 10187 log.cc:824] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:08.578007 10187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac: No bootstrap required, opened a new log
I20260924 07:12:08.581753 10187 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260924 07:12:08.582020 10187 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:08.582087 10187 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f8ea67eddb4e8985123424074c76ac, State: Initialized, Role: FOLLOWER
I20260924 07:12:08.582517 10187 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260924 07:12:08.582647 10187 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:08.582712 10187 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:08.582813 10187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:08.583811 10187 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260924 07:12:08.584158 10187 leader_election.cc:302] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f8ea67eddb4e8985123424074c76ac; no voters:
I20260924 07:12:08.584467 10187 leader_election.cc:288] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:08.584573 10192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:08.584779 10192 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [term 1 LEADER]: Becoming Leader. State: Replica: a7f8ea67eddb4e8985123424074c76ac, State: Running, Role: LEADER
I20260924 07:12:08.585091 10192 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260924 07:12:08.585676 10187 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:08.586696 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f8ea67eddb4e8985123424074c76ac. Latest consensus state: current_term: 1 leader_uuid: "a7f8ea67eddb4e8985123424074c76ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } } }
I20260924 07:12:08.586696 10193 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f8ea67eddb4e8985123424074c76ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f8ea67eddb4e8985123424074c76ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } } }
I20260924 07:12:08.586851 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:08.586853 10193 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:08.587232 10201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:08.590277 10201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:08.595506 10201 catalog_manager.cc:1347] Generated new cluster ID: a0d107085ac440a5a306f39d2f047a9d
I20260924 07:12:08.595577 10201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:08.618221 10201 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:08.618564 10201 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:08.635461 10201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f8ea67eddb4e8985123424074c76ac: Generated new TSK 0
I20260924 07:12:08.636258 10201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:12:08.642724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--builtin_ntp_servers=127.0.98.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:12:08.754395 10211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.754685 10211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.754729 10211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.759166 10211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.759238 10211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.759301 10211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:08.764590 10211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:08.765759 10211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.767199 10211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.776808 10219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.776805 10216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.776998 10211 server_base.cc:1034] running on GCE node
W20260924 07:12:08.776808 10217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.777434 10211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.778101 10211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.779276 10211 hybrid_clock.cc:648] HybridClock initialized: now 1790233928779247 us; error 41 us; skew 500 ppm
I20260924 07:12:08.781191 10211 webserver.cc:460] Webserver started at http://127.0.98.1:38873/ using document root <none> and password file <none>
I20260924 07:12:08.781780 10211 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.781926 10211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.782140 10211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.784320 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "72d7902e70ca4f148ff6ff800729535c"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "7b60f8dba1e199c084d37fbea256c84e"
server_key_iv: "4875a37d3e5379a0d1e14e2967292605"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.784893 10211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72d7902e70ca4f148ff6ff800729535c"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "7b60f8dba1e199c084d37fbea256c84e"
server_key_iv: "4875a37d3e5379a0d1e14e2967292605"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.788797 10211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:12:08.791342 10225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.792677 10211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:08.792786 10211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "72d7902e70ca4f148ff6ff800729535c"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "7b60f8dba1e199c084d37fbea256c84e"
server_key_iv: "4875a37d3e5379a0d1e14e2967292605"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.792936 10211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.797535 10211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.798148 10211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.798310 10211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.798848 10211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:08.799800 10211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:08.799856 10211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.799909 10211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:08.799932 10211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.808534 10211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42315
I20260924 07:12:08.808626 10338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42315 every 8 connection(s)
I20260924 07:12:08.809489 10211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:08.815614 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.811462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34393 (local address 127.0.98.62:43647)
0924 07:12:08.811701 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.811704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.812698 (+ 994us) server_negotiation.cc:407] Connection header received
0924 07:12:08.813979 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.813982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.814021 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.814104 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.815180 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.815183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.815185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:08.815196 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.815215 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:08.815223 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.815303 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.815462 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.815503 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 07:12:08.816497 10341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.811863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43647 (local address 127.0.98.1:34393)
0924 07:12:08.812540 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:08.812582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:08.813750 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:08.814261 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:08.814267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:08.814359 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:08.814900 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:08.814902 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:08.814946 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:08.815017 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:08.815293 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:08.815297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:08.815301 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:08.815501 (+ 200us) client_negotiation.cc:240] Negotiation successful
0924 07:12:08.815707 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260924 07:12:08.817885 10339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43647
I20260924 07:12:08.818113 10339 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:08.818514 10339 heartbeater.cc:511] Master 127.0.98.62:43647 requested a full tablet report, sending...
I20260924 07:12:08.818856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10211
I20260924 07:12:08.818956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:08.819231 392 external_mini_cluster.cc:1428] Setting key 514ad2f18bcbb3eaaef95594887ce264
I20260924 07:12:08.820313 10151 ts_manager.cc:194] Registered new tserver with Master: 72d7902e70ca4f148ff6ff800729535c (127.0.98.1:42315)
I20260924 07:12:08.821353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--builtin_ntp_servers=127.0.98.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:12:08.932128 10342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.932389 10342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.932437 10342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.936456 10342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.936519 10342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.936578 10342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:08.941416 10342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:08.942528 10342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.943971 10342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:12:08.953734 10342 server_base.cc:1034] running on GCE node
W20260924 07:12:08.953692 10347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.953692 10350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.953696 10348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.954481 10342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.955148 10342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.956312 10342 hybrid_clock.cc:648] HybridClock initialized: now 1790233928956274 us; error 51 us; skew 500 ppm
I20260924 07:12:08.958313 10342 webserver.cc:460] Webserver started at http://127.0.98.2:39513/ using document root <none> and password file <none>
I20260924 07:12:08.958877 10342 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.958947 10342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.959149 10342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.961324 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "063b091979204b04aa4e9c02031186fc"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "e174fc221215c0b7b39672ad062500b8"
server_key_iv: "af27347bde851e1225508b5732fac3cb"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.961882 10342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "063b091979204b04aa4e9c02031186fc"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "e174fc221215c0b7b39672ad062500b8"
server_key_iv: "af27347bde851e1225508b5732fac3cb"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.965723 10342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:12:08.968210 10356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.969439 10342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:08.969545 10342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "063b091979204b04aa4e9c02031186fc"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-cnv3"
server_key: "e174fc221215c0b7b39672ad062500b8"
server_key_iv: "af27347bde851e1225508b5732fac3cb"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.969641 10342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.974112 10342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.974676 10342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.974822 10342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.975322 10342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:08.976187 10342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:08.976233 10342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.976281 10342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:08.976307 10342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.984419 10342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36869
I20260924 07:12:08.984566 10469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36869 every 8 connection(s)
I20260924 07:12:08.985325 10342 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:08.987861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10342
I20260924 07:12:08.987963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:08.988242 392 external_mini_cluster.cc:1428] Setting key cb5ed608383fea9d99bc58872c0f2a92
I20260924 07:12:08.990275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--builtin_ntp_servers=127.0.98.20:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 07:12:08.991976 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.987227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56075 (local address 127.0.98.62:43647)
0924 07:12:08.987406 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.987410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.988685 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 07:12:08.990145 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.990147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.990180 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.990240 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.991513 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.991517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.991520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:08.991535 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.991562 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:08.991573 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.991669 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.991798 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.991839 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:12:08.992836 10472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.987718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43647 (local address 127.0.98.2:56075)
0924 07:12:08.988509 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:08.988563 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:08.989925 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:08.990448 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:08.990456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:08.990568 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:08.991175 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:08.991177 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:08.991233 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:08.991328 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:08.991665 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:08.991673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:08.991679 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:12:08.991836 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:12:08.992038 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260924 07:12:08.993924 10470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43647
I20260924 07:12:08.994170 10470 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:08.994656 10470 heartbeater.cc:511] Master 127.0.98.62:43647 requested a full tablet report, sending...
I20260924 07:12:08.995756 10151 ts_manager.cc:194] Registered new tserver with Master: 063b091979204b04aa4e9c02031186fc (127.0.98.2:36869)
W20260924 07:12:09.101989 10473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.102274 10473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.102313 10473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.106400 10473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.106462 10473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.106523 10473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:09.111493 10473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:09.112636 10473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.114040 10473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.123687 10481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.123708 10479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.123687 10478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.124053 10473 server_base.cc:1034] running on GCE node
I20260924 07:12:09.124439 10473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.125105 10473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.126281 10473 hybrid_clock.cc:648] HybridClock initialized: now 1790233929126258 us; error 38 us; skew 500 ppm
I20260924 07:12:09.128276 10473 webserver.cc:460] Webserver started at http://127.0.98.3:35107/ using document root <none> and password file <none>
I20260924 07:12:09.128844 10473 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.128919 10473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.129117 10473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:09.131163 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff8a7ae9364a4f49a3e6636d384824c1"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-cnv3"
server_key: "db7d21738246afbd3f2581463348e924"
server_key_iv: "1493d2d26befdd90737cdb3ce2077cd7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.131731 10473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff8a7ae9364a4f49a3e6636d384824c1"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-cnv3"
server_key: "db7d21738246afbd3f2581463348e924"
server_key_iv: "1493d2d26befdd90737cdb3ce2077cd7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.135329 10473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:12:09.137738 10487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.139114 10473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:09.139230 10473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "ff8a7ae9364a4f49a3e6636d384824c1"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-cnv3"
server_key: "db7d21738246afbd3f2581463348e924"
server_key_iv: "1493d2d26befdd90737cdb3ce2077cd7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.139331 10473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.143781 10473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.144378 10473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.144528 10473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.145041 10473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.145915 10473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:09.145959 10473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.146008 10473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:09.146034 10473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.154393 10473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39889
I20260924 07:12:09.154467 10600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39889 every 8 connection(s)
I20260924 07:12:09.155313 10473 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:09.156934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10473
I20260924 07:12:09.157034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:09.157305 392 external_mini_cluster.cc:1428] Setting key f1570b59a86c8597150fab6c1962c30e
I20260924 07:12:09.161532 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.157188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38593 (local address 127.0.98.62:43647)
0924 07:12:09.157338 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.157343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.158552 (+ 1209us) server_negotiation.cc:407] Connection header received
0924 07:12:09.159928 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.159930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.159967 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.160037 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.161113 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.161117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.161120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:12:09.161130 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.161150 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:12:09.161159 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.161230 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.161357 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.161405 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:12:09.162417 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.157651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43647 (local address 127.0.98.3:38593)
0924 07:12:09.158399 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.158451 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.159663 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.160183 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.160190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.160279 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:09.160796 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:12:09.160798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:09.160838 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:12:09.160969 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:09.161223 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:09.161231 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:09.161236 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.161389 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.161641 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":122,"threads_started":1}
I20260924 07:12:09.163431 10601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43647
I20260924 07:12:09.163625 10601 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:09.164036 10601 heartbeater.cc:511] Master 127.0.98.62:43647 requested a full tablet report, sending...
I20260924 07:12:09.164989 10151 ts_manager.cc:194] Registered new tserver with Master: ff8a7ae9364a4f49a3e6636d384824c1 (127.0.98.3:39889)
I20260924 07:12:09.170825 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:12:09.173525 10190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:09.172632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.98.62:43647)
0924 07:12:09.172741 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.172746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.172957 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:12:09.173084 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.173087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.173118 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.173177 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.173404 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33708: BlockingRecv error: recv got EOF from 127.0.0.1:33708 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:12:09.173748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10211
I20260924 07:12:09.179538 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10342
I20260924 07:12:09.184134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10473
I20260924 07:12:09.189101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10120
2026-09-24T07:12:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (814 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:09 dist-test-slave-cnv3 krb5kdc[10611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:09 dist-test-slave-cnv3 krb5kdc[10611](info): set up 2 sockets
Sep 24 07:12:09 dist-test-slave-cnv3 krb5kdc[10611](info): commencing operation
krb5kdc: starting...
W20260924 07:12:10.152155 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:10.270700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40467
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:10.377657 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.377926 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.377965 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.381937 10624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:10.381999 10624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.382017 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.382032 10624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:10.382046 10624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:10.387356 10624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:10.388458 10624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.389441 10624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.397758 10632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.397784 10629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.397758 10630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.397929 10624 server_base.cc:1034] running on GCE node
I20260924 07:12:10.398698 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.399912 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.401086 10624 hybrid_clock.cc:648] HybridClock initialized: now 1790233930401066 us; error 33 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.403820 10624 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:10.404727 10624 webserver.cc:460] Webserver started at http://127.0.98.62:35683/ using document root <none> and password file <none>
I20260924 07:12:10.405264 10624 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.405330 10624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.405519 10624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:10.407665 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "0a495319aad0455db4f003e31db05edc"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "73b277bad8e134f84956861219f570d0"
server_key_iv: "3cfe54a71b1d365ff5d163aaaf52e420"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.408206 10624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a495319aad0455db4f003e31db05edc"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "73b277bad8e134f84956861219f570d0"
server_key_iv: "3cfe54a71b1d365ff5d163aaaf52e420"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.411873 10624 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:12:10.414243 10639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.415601 10624 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:10.415716 10624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "0a495319aad0455db4f003e31db05edc"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "73b277bad8e134f84956861219f570d0"
server_key_iv: "3cfe54a71b1d365ff5d163aaaf52e420"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.415813 10624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.433205 10624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.436492 10624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.436679 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.443243 10624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40467
I20260924 07:12:10.443512 10691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40467 every 8 connection(s)
I20260924 07:12:10.444195 10624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:10.446806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10624
I20260924 07:12:10.446909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:10.447042 10692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:10.447170 392 external_mini_cluster.cc:1428] Setting key 59985d90f2cb1ed2637cac3833df5afa
I20260924 07:12:10.454310 10692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: Bootstrap starting.
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:10.456727 10692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:10.457616 10692 log.cc:824] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:10.459905 10692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: No bootstrap required, opened a new log
I20260924 07:12:10.461824 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.448594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.0.98.62:40467)
0924 07:12:10.449007 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.449025 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.449069 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:12:10.449759 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.449781 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.450216 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.450621 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:10.451197 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.452669 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.453495 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.454169 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.454322 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.456732 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:10.456738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:10.456756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:10.456790 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:10.458640 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.459045 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.459051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.459057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.459192 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.459444 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.459447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.459448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.459953 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:10.460133 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.460534 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.460823 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":99,"threads_started":1}
I20260924 07:12:10.463274 10692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260924 07:12:10.463546 10692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:10.463603 10692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a495319aad0455db4f003e31db05edc, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.464035 10692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260924 07:12:10.464200 10692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:10.464262 10692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:10.464350 10692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:10.465246 10692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260924 07:12:10.465564 10692 leader_election.cc:302] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a495319aad0455db4f003e31db05edc; no voters:
I20260924 07:12:10.465860 10692 leader_election.cc:288] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:10.465952 10697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:10.466183 10697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [term 1 LEADER]: Becoming Leader. State: Replica: 0a495319aad0455db4f003e31db05edc, State: Running, Role: LEADER
I20260924 07:12:10.466462 10697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260924 07:12:10.467053 10692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:10.468041 10698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a495319aad0455db4f003e31db05edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } }
I20260924 07:12:10.468166 10698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:10.468417 10699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a495319aad0455db4f003e31db05edc. Latest consensus state: current_term: 1 leader_uuid: "0a495319aad0455db4f003e31db05edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a495319aad0455db4f003e31db05edc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } }
I20260924 07:12:10.468501 10699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:10.468627 10706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:10.471307 10706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:10.476611 10706 catalog_manager.cc:1347] Generated new cluster ID: 6375a7531faa4297a97b6e3e3321178b
I20260924 07:12:10.476703 10706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:10.490845 10706 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:10.492267 10706 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:10.500159 10706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: Generated new TSK 0
I20260924 07:12:10.501036 10706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:12:10.535300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:10.647959 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.648263 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.648307 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.652474 10718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.652542 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.652603 10718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:10.658002 10718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:10.659168 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.660213 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.670346 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.670363 10726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.670329 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.670603 10718 server_base.cc:1034] running on GCE node
I20260924 07:12:10.671029 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.671707 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.672881 10718 hybrid_clock.cc:648] HybridClock initialized: now 1790233930672817 us; error 77 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.679143 10718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:10.680204 10718 webserver.cc:460] Webserver started at http://127.0.98.1:34453/ using document root <none> and password file <none>
I20260924 07:12:10.680811 10718 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.680884 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.681078 10718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:10.683221 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bada4421e6d48daa812501092d34aca"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "72e123671bc23fa4bf0f08407ef3a4c7"
server_key_iv: "b0b6861cddddcbc878cd138c441e5f58"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.683830 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bada4421e6d48daa812501092d34aca"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "72e123671bc23fa4bf0f08407ef3a4c7"
server_key_iv: "b0b6861cddddcbc878cd138c441e5f58"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.687904 10718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:12:10.690492 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.691927 10718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:10.692078 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "1bada4421e6d48daa812501092d34aca"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "72e123671bc23fa4bf0f08407ef3a4c7"
server_key_iv: "b0b6861cddddcbc878cd138c441e5f58"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.692188 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.707033 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.710276 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.710460 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.711048 10718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:10.711910 10718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:10.711966 10718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.712023 10718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:10.712051 10718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.720697 10718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43177
I20260924 07:12:10.720754 10846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43177 every 8 connection(s)
I20260924 07:12:10.721616 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:10.731878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10718
I20260924 07:12:10.731981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:10.732234 392 external_mini_cluster.cc:1428] Setting key 58cb094d31e8158e9525226a54d98eed
I20260924 07:12:10.735526 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.723529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35195 (local address 127.0.98.62:40467)
0924 07:12:10.723676 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.723680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.724786 (+ 1106us) server_negotiation.cc:407] Connection header received
0924 07:12:10.725714 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.725717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.725763 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.725836 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:10.727146 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.727749 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.729001 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.729256 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.729344 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.731916 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:10.731919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:10.731921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:10.731949 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:10.733563 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.734034 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.734037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.734040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.734122 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.734440 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.734443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.734445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.734626 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:10.734718 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.735208 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.735330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:12:10.736392 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.723975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:35195)
0924 07:12:10.724640 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.724683 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.725515 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.726027 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.726033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.726450 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:10.726958 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.726972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.727881 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.727883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.728874 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.728883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.729424 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.729427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.729593 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.730203 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:10.730220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:10.731730 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:10.733663 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.733670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.733688 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.733943 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.734210 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.734213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.734216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.734357 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.734721 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:10.734728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:10.734985 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.735154 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.735533 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":110,"threads_started":1}
I20260924 07:12:10.737793 10847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260924 07:12:10.738049 10847 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:10.738543 10847 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260924 07:12:10.740149 10656 ts_manager.cc:194] Registered new tserver with Master: 1bada4421e6d48daa812501092d34aca (127.0.98.1:43177)
I20260924 07:12:10.741319 10656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35195
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:12:10.760619 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:10.873806 10852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.874085 10852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.874127 10852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.878352 10852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.878425 10852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.878485 10852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:10.883912 10852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:10.884953 10852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.885949 10852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.895466 10857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.895576 10860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.895606 10852 server_base.cc:1034] running on GCE node
W20260924 07:12:10.895650 10858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.896243 10852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.896919 10852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.898079 10852 hybrid_clock.cc:648] HybridClock initialized: now 1790233930898053 us; error 38 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.900795 10852 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:10.901603 10852 webserver.cc:460] Webserver started at http://127.0.98.2:39949/ using document root <none> and password file <none>
I20260924 07:12:10.902114 10852 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.902171 10852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.902328 10852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:10.904374 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca02338665fd4417ab32c1892380b10f"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "ee21929ad81d40c0b0674c71acf0d96f"
server_key_iv: "e87d2a0a99111cd960510060940ba3c0"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.904873 10852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca02338665fd4417ab32c1892380b10f"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "ee21929ad81d40c0b0674c71acf0d96f"
server_key_iv: "e87d2a0a99111cd960510060940ba3c0"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.908509 10852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:12:10.910815 10867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.912062 10852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:12:10.912158 10852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "ca02338665fd4417ab32c1892380b10f"
format_stamp: "Formatted at 2026-09-24 07:12:10 on dist-test-slave-cnv3"
server_key: "ee21929ad81d40c0b0674c71acf0d96f"
server_key_iv: "e87d2a0a99111cd960510060940ba3c0"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.912261 10852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.952512 10852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.955349 10852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.955547 10852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.956147 10852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:10.956980 10852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:10.957026 10852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.957079 10852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:10.957108 10852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.965899 10852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42061
I20260924 07:12:10.965922 10980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42061 every 8 connection(s)
I20260924 07:12:10.966764 10852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:10.967253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10852
I20260924 07:12:10.967348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:10.967646 392 external_mini_cluster.cc:1428] Setting key c40bb8b0f2376aea9a4d665b86daf345
Sep 24 07:12:10 dist-test-slave-cnv3 krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:12:10.981290 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.968730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44321 (local address 127.0.98.62:40467)
0924 07:12:10.968859 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.968871 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.970151 (+ 1280us) server_negotiation.cc:407] Connection header received
0924 07:12:10.971158 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.971162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.971216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.971303 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:10.972550 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.973068 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.974729 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.974939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.975014 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.977835 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:10.977839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:10.977841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:10.977865 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:10.979145 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.979673 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.979676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.979677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.979785 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.980050 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.980052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.980053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.980287 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:10.980359 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.980991 (+ 632us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.981123 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:12:10.982146 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.969214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:44321)
0924 07:12:10.969958 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.970013 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.970950 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.971474 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.971481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.971914 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:10.972407 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.972420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.973202 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.973206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.974618 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.974629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.975083 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.975088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.975301 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.975923 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:10.975943 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:10.977663 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:10.979245 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.979250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.979263 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.979556 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.979851 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.979853 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.979854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.979981 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.980432 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:10.980439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:10.980743 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.980910 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.981195 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":109,"threads_started":1}
I20260924 07:12:10.983439 10981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260924 07:12:10.983665 10981 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:10.984138 10981 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260924 07:12:10.985137 10656 ts_manager.cc:194] Registered new tserver with Master: ca02338665fd4417ab32c1892380b10f (127.0.98.2:42061)
I20260924 07:12:10.985677 10656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:12:10.996779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:11.110561 10986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:11.110857 10986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:11.110899 10986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:11.115064 10986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:11.115144 10986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:11.115203 10986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:11.120776 10986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:11.121901 10986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:11.122957 10986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:11.133024 10991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:11.133028 10992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:11.133044 10994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:11.133334 10986 server_base.cc:1034] running on GCE node
I20260924 07:12:11.133822 10986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:11.134534 10986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:11.135725 10986 hybrid_clock.cc:648] HybridClock initialized: now 1790233931135698 us; error 40 us; skew 500 ppm
Sep 24 07:12:11 dist-test-slave-cnv3 krb5kdc[10611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:11.138602 10986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:11.139638 10986 webserver.cc:460] Webserver started at http://127.0.98.3:45073/ using document root <none> and password file <none>
I20260924 07:12:11.140193 10986 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:11.140252 10986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:11.140447 10986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:11.142495 10986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ee45334c3b14a519c2303882350b589"
format_stamp: "Formatted at 2026-09-24 07:12:11 on dist-test-slave-cnv3"
server_key: "02bceb96e1f38bc4e26fc6200432f62f"
server_key_iv: "4d00901d090337b9dc2e8053d23add58"
server_key_version: "encryptionkey@0"
I20260924 07:12:11.143044 10986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee45334c3b14a519c2303882350b589"
format_stamp: "Formatted at 2026-09-24 07:12:11 on dist-test-slave-cnv3"
server_key: "02bceb96e1f38bc4e26fc6200432f62f"
server_key_iv: "4d00901d090337b9dc2e8053d23add58"
server_key_version: "encryptionkey@0"
I20260924 07:12:11.146781 10986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 07:12:11.149353 11001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:11.150676 10986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:11.150774 10986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "5ee45334c3b14a519c2303882350b589"
format_stamp: "Formatted at 2026-09-24 07:12:11 on dist-test-slave-cnv3"
server_key: "02bceb96e1f38bc4e26fc6200432f62f"
server_key_iv: "4d00901d090337b9dc2e8053d23add58"
server_key_version: "encryptionkey@0"
I20260924 07:12:11.150882 10986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:11.173067 10986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:11.176093 10986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:11.176276 10986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:11.176862 10986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:11.177673 10986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:11.177723 10986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:11.177765 10986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:11.177795 10986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:11.186306 10986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34355
I20260924 07:12:11.186337 11114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34355 every 8 connection(s)
I20260924 07:12:11.187217 10986 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:11.193367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 10986
I20260924 07:12:11.193478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:11.193753 392 external_mini_cluster.cc:1428] Setting key 2896c1bccbd9a1eec845ec0a2e18dc05
Sep 24 07:12:11 dist-test-slave-cnv3 krb5kdc[10611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:11.200532 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.189033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53605 (local address 127.0.98.62:40467)
0924 07:12:11.189158 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.189161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.190249 (+ 1088us) server_negotiation.cc:407] Connection header received
0924 07:12:11.191151 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.191154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.191208 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.191277 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:11.192500 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.193040 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.194216 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.194479 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.194604 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.197170 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:11.197174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:11.197177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:11.197205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:11.198479 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.198932 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.198934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.198935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.199007 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.199452 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.199455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.199456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.199638 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:11.199735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.200313 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.200414 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:12:11.201327 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.189449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:53605)
0924 07:12:11.190111 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.190154 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.190956 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.191505 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.191513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.191862 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:11.192349 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.192362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.193157 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.193160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.194097 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.194107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.194631 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.194636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.194829 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.195427 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:11.195447 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:11.197017 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:11.198581 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.198586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.198600 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.198858 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.199154 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.199157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.199158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.199311 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.199759 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:11.199764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:11.200014 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.200252 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.200517 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":110,"threads_started":1}
I20260924 07:12:11.202442 11115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260924 07:12:11.202668 11115 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:11.203145 11115 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260924 07:12:11.204264 10656 ts_manager.cc:194] Registered new tserver with Master: 5ee45334c3b14a519c2303882350b589 (127.0.98.3:34355)
I20260924 07:12:11.204811 10656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53605
I20260924 07:12:11.207495 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:11.215961 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.209288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.0.98.62:40467)
0924 07:12:11.209411 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.209415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.209488 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:12:11.209644 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.209647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.209688 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.209751 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:11.210468 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.210978 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.211835 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.212058 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.212142 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.213382 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:11.213386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:11.213389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:11.213414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:11.214425 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.214786 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.214790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.214792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.214868 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.215095 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.215099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.215100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.215326 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:11.215473 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.215697 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.215808 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:12:11.218696 10656 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:11.221814 10656 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:11.236238 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.231497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.98.2:42061)
0924 07:12:11.231863 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.231876 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.231896 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:12:11.231962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.231967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.232218 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.232348 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:11.232684 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233386 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.234941 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.235593 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.235700 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.235854 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.235991 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.236086 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":90,"threads_started":1}
I20260924 07:12:11.236228 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.231052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.0.1:43736)
0924 07:12:11.231325 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.231335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.231454 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.232491 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.232494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.232509 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:11.232579 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.232585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233499 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.233503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.234814 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.234825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.235708 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.235711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.235812 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.235846 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.235957 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.236081 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260924 07:12:11.236635 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.230352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.0.1:57100)
0924 07:12:11.231282 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.231296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.231454 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.232210 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.232216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.232248 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:11.232401 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.232406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233841 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.233844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.234981 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.234989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.236198 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.236202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.236334 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.236358 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.236427 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.236493 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":96,"spinlock_wait_cycles":16768,"thread_start_us":79,"threads_started":1}
I20260924 07:12:11.236712 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.230930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.0.98.1:43177)
0924 07:12:11.231728 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.231745 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.231766 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:12:11.231834 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.231839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.232037 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.232197 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:11.232502 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233704 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.235097 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.235905 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.236004 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.236138 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.236465 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.236563 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":109,"threads_started":1}
I20260924 07:12:11.237784 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.230480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.0.98.3:34355)
0924 07:12:11.230788 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.230799 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.231304 (+ 505us) server_negotiation.cc:407] Connection header received
0924 07:12:11.231662 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.231668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.231884 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.232006 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:11.232554 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233709 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.235041 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.235907 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.236004 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.236138 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.237556 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.237652 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260924 07:12:11.238179 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.230330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34355 (local address 127.0.0.1:50350)
0924 07:12:11.231162 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.231213 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.231455 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.232211 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.232216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.232243 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:11.232411 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.232418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.233870 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.233873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.234924 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.234933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.236198 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.236202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.237393 (+ 1191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.237425 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.237960 (+ 535us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.238045 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":76,"spinlock_wait_cycles":5504,"thread_start_us":89,"threads_started":1}
I20260924 07:12:11.239338 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 7d618cddf6f14e268362282b65121534 (DEFAULT_TABLE table=test-table [id=b500579922ac4b279ab91c5a16e20eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:11.239354 10915 tablet_service.cc:1467] Processing CreateTablet for tablet 7d618cddf6f14e268362282b65121534 (DEFAULT_TABLE table=test-table [id=b500579922ac4b279ab91c5a16e20eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:11.240350 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d618cddf6f14e268362282b65121534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:11.240378 10915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d618cddf6f14e268362282b65121534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:11.240852 11049 tablet_service.cc:1467] Processing CreateTablet for tablet 7d618cddf6f14e268362282b65121534 (DEFAULT_TABLE table=test-table [id=b500579922ac4b279ab91c5a16e20eda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:11.241859 11049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d618cddf6f14e268362282b65121534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:11.247318 11133 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Bootstrap starting.
I20260924 07:12:11.247617 11132 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Bootstrap starting.
I20260924 07:12:11.248354 11134 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Bootstrap starting.
I20260924 07:12:11.249732 11132 tablet_bootstrap.cc:654] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:11.250133 11134 tablet_bootstrap.cc:654] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:11.250139 11133 tablet_bootstrap.cc:654] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:11.250630 11132 log.cc:824] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:11.250912 11133 log.cc:824] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:11.251122 11134 log.cc:824] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:11.252851 11133 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: No bootstrap required, opened a new log
I20260924 07:12:11.252846 11132 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: No bootstrap required, opened a new log
I20260924 07:12:11.253039 11133 ts_tablet_manager.cc:1397] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:12:11.253042 11132 ts_tablet_manager.cc:1397] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:12:11.253134 11134 tablet_bootstrap.cc:492] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: No bootstrap required, opened a new log
I20260924 07:12:11.253299 11134 ts_tablet_manager.cc:1397] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:12:11.256143 11133 raft_consensus.cc:348] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.256386 11133 raft_consensus.cc:374] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:11.256438 11133 raft_consensus.cc:729] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca02338665fd4417ab32c1892380b10f, State: Initialized, Role: FOLLOWER
I20260924 07:12:11.256868 11133 consensus_queue.cc:260] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.257026 11134 raft_consensus.cc:348] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.257340 11134 raft_consensus.cc:374] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:11.257398 11134 raft_consensus.cc:729] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee45334c3b14a519c2303882350b589, State: Initialized, Role: FOLLOWER
I20260924 07:12:11.257627 10981 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
I20260924 07:12:11.257539 11132 raft_consensus.cc:348] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.257764 11132 raft_consensus.cc:374] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:11.257805 11132 raft_consensus.cc:729] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bada4421e6d48daa812501092d34aca, State: Initialized, Role: FOLLOWER
I20260924 07:12:11.257900 11133 ts_tablet_manager.cc:1428] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:12:11.257834 11134 consensus_queue.cc:260] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.258201 11132 consensus_queue.cc:260] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.258821 11134 ts_tablet_manager.cc:1428] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260924 07:12:11.259162 11115 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
I20260924 07:12:11.259554 11132 ts_tablet_manager.cc:1428] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 07:12:11.259871 10847 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
W20260924 07:12:11.438247 11116 tablet.cc:2369] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:11.441020 11140 raft_consensus.cc:482] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:11.441218 11140 raft_consensus.cc:504] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.442327 11140 leader_election.cc:288] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee45334c3b14a519c2303882350b589 (127.0.98.3:34355), ca02338665fd4417ab32c1892380b10f (127.0.98.2:42061)
I20260924 07:12:11.447117 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.442771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60151 (local address 127.0.98.2:42061)
0924 07:12:11.442885 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.442889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.443202 (+ 313us) server_negotiation.cc:407] Connection header received
0924 07:12:11.443442 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.443445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.443487 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.443548 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:11.443945 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.444663 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.445914 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.446502 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.446615 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.446685 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.446885 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.446969 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:12:11.447276 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.442691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47613 (local address 127.0.98.3:34355)
0924 07:12:11.442855 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.442860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.443219 (+ 359us) server_negotiation.cc:407] Connection header received
0924 07:12:11.443406 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.443410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.443465 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.443555 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:11.443840 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.444624 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.446145 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.446730 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.446831 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.446886 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.447088 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.447159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 07:12:11.447316 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.442641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34355 (local address 127.0.98.1:47613)
0924 07:12:11.443129 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.443141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.443276 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.443581 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.443584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.443601 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:11.443722 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.443727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.444755 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.444759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.446015 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.446026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.446854 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.446858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.446965 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.446985 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.447052 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.447139 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":33,"spinlock_wait_cycles":384,"thread_start_us":111,"threads_started":1}
I20260924 07:12:11.447700 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.442629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42061 (local address 127.0.98.1:60151)
0924 07:12:11.443014 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.443031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.443276 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.443679 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.443683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.443702 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:11.443826 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.443832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.444779 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.444782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.445753 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.445765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.446634 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.446637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.446733 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.446760 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.447503 (+ 743us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.447572 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":84,"spinlock_wait_cycles":5248,"thread_start_us":124,"threads_started":1}
I20260924 07:12:11.448007 11069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d618cddf6f14e268362282b65121534" candidate_uuid: "1bada4421e6d48daa812501092d34aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee45334c3b14a519c2303882350b589" is_pre_election: true
I20260924 07:12:11.448338 11069 raft_consensus.cc:2454] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bada4421e6d48daa812501092d34aca in term 0.
I20260924 07:12:11.448310 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d618cddf6f14e268362282b65121534" candidate_uuid: "1bada4421e6d48daa812501092d34aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca02338665fd4417ab32c1892380b10f" is_pre_election: true
I20260924 07:12:11.448576 10935 raft_consensus.cc:2454] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bada4421e6d48daa812501092d34aca in term 0.
I20260924 07:12:11.448794 10737 leader_election.cc:302] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bada4421e6d48daa812501092d34aca, 5ee45334c3b14a519c2303882350b589; no voters:
I20260924 07:12:11.449100 11140 raft_consensus.cc:2781] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:11.449208 11140 raft_consensus.cc:482] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:11.449246 11140 raft_consensus.cc:3037] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:11.450480 11140 raft_consensus.cc:504] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.450857 11140 leader_election.cc:288] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [CANDIDATE]: Term 1 election: Requested vote from peers 5ee45334c3b14a519c2303882350b589 (127.0.98.3:34355), ca02338665fd4417ab32c1892380b10f (127.0.98.2:42061)
I20260924 07:12:11.451184 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d618cddf6f14e268362282b65121534" candidate_uuid: "1bada4421e6d48daa812501092d34aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca02338665fd4417ab32c1892380b10f"
I20260924 07:12:11.451184 11069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d618cddf6f14e268362282b65121534" candidate_uuid: "1bada4421e6d48daa812501092d34aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee45334c3b14a519c2303882350b589"
I20260924 07:12:11.451339 10935 raft_consensus.cc:3037] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:11.451361 11069 raft_consensus.cc:3037] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:11.452607 11069 raft_consensus.cc:2454] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bada4421e6d48daa812501092d34aca in term 1.
I20260924 07:12:11.452607 10935 raft_consensus.cc:2454] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bada4421e6d48daa812501092d34aca in term 1.
I20260924 07:12:11.452908 10735 leader_election.cc:302] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bada4421e6d48daa812501092d34aca, ca02338665fd4417ab32c1892380b10f; no voters:
I20260924 07:12:11.453082 11140 raft_consensus.cc:2781] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:11.453321 11140 raft_consensus.cc:686] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [term 1 LEADER]: Becoming Leader. State: Replica: 1bada4421e6d48daa812501092d34aca, State: Running, Role: LEADER
I20260924 07:12:11.453662 11140 consensus_queue.cc:237] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } }
I20260924 07:12:11.456983 10652 catalog_manager.cc:5495] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bada4421e6d48daa812501092d34aca (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1bada4421e6d48daa812501092d34aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bada4421e6d48daa812501092d34aca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 } health_report { overall_health: UNKNOWN } } }
W20260924 07:12:11.468030 10982 tablet.cc:2369] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:11.471119 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.467651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57108 (local address 127.0.98.1:43177)
0924 07:12:11.467781 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.467784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.467797 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:12:11.467880 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.467883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.467950 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.468060 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:11.468371 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.468966 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.469864 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.470081 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.470155 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.470429 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:11.470836 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:11.470923 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.470966 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.471014 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 07:12:11.472872 10848 tablet.cc:2369] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:11.476861 11069 raft_consensus.cc:1261] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 1 FOLLOWER]: Refusing update from remote peer 1bada4421e6d48daa812501092d34aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:11.476892 10935 raft_consensus.cc:1261] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 1 FOLLOWER]: Refusing update from remote peer 1bada4421e6d48daa812501092d34aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:11.477476 11140 consensus_queue.cc:1035] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca02338665fd4417ab32c1892380b10f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:11.477679 11143 consensus_queue.cc:1035] T 7d618cddf6f14e268362282b65121534 P 1bada4421e6d48daa812501092d34aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee45334c3b14a519c2303882350b589" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:11.484927 11146 mvcc.cc:205] Tried to move back new op lower bound from 7332798183321866240 to 7332798183237013504. Current Snapshot: MvccSnapshot[applied={T|T < 7332798183321866240}]
I20260924 07:12:11.485556 11147 mvcc.cc:205] Tried to move back new op lower bound from 7332798183321866240 to 7332798183237013504. Current Snapshot: MvccSnapshot[applied={T|T < 7332798183321866240}]
I20260924 07:12:11.493441 10652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:11.493620 10652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:11.496043 10652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a495319aad0455db4f003e31db05edc: Sending DeleteTablet for 3 replicas of tablet 7d618cddf6f14e268362282b65121534
I20260924 07:12:11.496752 11049 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d618cddf6f14e268362282b65121534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50350
I20260924 07:12:11.496745 10915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d618cddf6f14e268362282b65121534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43736
I20260924 07:12:11.497249 10781 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d618cddf6f14e268362282b65121534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57100
I20260924 07:12:11.497391 11157 tablet_replica.cc:330] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: stopping tablet replica
I20260924 07:12:11.497694 11156 tablet_replica.cc:330] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: stopping tablet replica
I20260924 07:12:11.497712 11157 raft_consensus.cc:2229] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:11.498018 11157 raft_consensus.cc:2258] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:11.498253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10718
I20260924 07:12:11.499625 11156 raft_consensus.cc:2229] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:11.499714 11157 ts_tablet_manager.cc:1905] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:11.499949 11156 raft_consensus.cc:2258] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:11.501374 11156 ts_tablet_manager.cc:1905] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:11.502125 11157 ts_tablet_manager.cc:1918] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:11.502190 11157 log.cc:1196] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/7d618cddf6f14e268362282b65121534
I20260924 07:12:11.502403 11157 ts_tablet_manager.cc:1939] T 7d618cddf6f14e268362282b65121534 P 5ee45334c3b14a519c2303882350b589: Deleting consensus metadata
I20260924 07:12:11.503535 10643 catalog_manager.cc:4841] TS 5ee45334c3b14a519c2303882350b589 (127.0.98.3:34355): tablet 7d618cddf6f14e268362282b65121534 (table test-table [id=b500579922ac4b279ab91c5a16e20eda]) successfully deleted
I20260924 07:12:11.503952 11156 ts_tablet_manager.cc:1918] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:11.504050 11156 log.cc:1196] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/7d618cddf6f14e268362282b65121534
I20260924 07:12:11.504415 11156 ts_tablet_manager.cc:1939] T 7d618cddf6f14e268362282b65121534 P ca02338665fd4417ab32c1892380b10f: Deleting consensus metadata
W20260924 07:12:11.505475 10642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43177 (error 108)
I20260924 07:12:11.505618 10641 catalog_manager.cc:4841] TS ca02338665fd4417ab32c1892380b10f (127.0.98.2:42061): tablet 7d618cddf6f14e268362282b65121534 (table test-table [id=b500579922ac4b279ab91c5a16e20eda]) successfully deleted
I20260924 07:12:11.506225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10852
I20260924 07:12:11.506732 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.506443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43177 (local address 127.0.0.1:57122)
0924 07:12:11.506595 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.506650 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260924 07:12:11.506889 10642 catalog_manager.cc:4584] TS 1bada4421e6d48daa812501092d34aca (127.0.98.1:43177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d618cddf6f14e268362282b65121534: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43177: connect: Connection refused (error 111)
I20260924 07:12:11.511571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10986
I20260924 07:12:11.516390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 10624
2026-09-24T07:12:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2328 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:11 dist-test-slave-cnv3 krb5kdc[11162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:11 dist-test-slave-cnv3 krb5kdc[11162](info): set up 2 sockets
Sep 24 07:12:11 dist-test-slave-cnv3 krb5kdc[11162](info): commencing operation
krb5kdc: starting...
W20260924 07:12:12.472177 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:12 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 07:12:12.640347 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasks9UDai/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/postgres -l logfile start
2026-09-24 07:12:15.246 UTC [11192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:12:15.246 UTC [11192] LOG: listening on IPv4 address "127.0.98.20", port 48263
2026-09-24 07:12:15.251 UTC [11192] LOG: listening on Unix socket "/tmp/.s.PGSQL.48263"
2026-09-24 07:12:15.275 UTC [11195] LOG: database system was shut down at 2026-09-24 07:12:13 UTC
2026-09-24 07:12:15.279 UTC [11192] LOG: database system is ready to accept connections
I20260924 07:12:16.143981 392 mini_postgres.cc:96] Postgres bound to 48263
2026-09-24 07:12:16.150 UTC [11203] FATAL: database "slave" does not exist
127.0.98.20:48263 - accepting connections
I20260924 07:12:16.150637 392 mini_ranger.cc:161] Starting Ranger...
I20260924 07:12:16.165508 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:12:16.690632 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:12:16.690716 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin
2026-09-24 07:12:16,950 [I] DB FLAVOR :POSTGRES
2026-09-24 07:12:16,950 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:12:16,951 [I] Checking connection..
2026-09-24 07:12:16,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:12:17,211 [I] Checking connection passed.
2026-09-24 07:12:17,211 [I] --------- Verifying version history table ---------
2026-09-24 07:12:17,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:12:17,471 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:12:17,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:12:17,730 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:12:17,730 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:12:17,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:12:17.970 UTC [11267] WARNING: there is no transaction in progress
2026-09-24 07:12:17,976 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:12:17,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:12:18,244 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:12:18,244 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:12:18,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:12:18,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:12:18,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cnv3.c.gcp-upstream.internal','N') ;"
2026-09-24 07:12:18,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:12:19,194 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:12:19,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:12:19,453 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:12:19,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:12:19,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:12:19,962 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:12:19,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:12:20.994 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.012 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.030 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.048 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.068 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.089 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.160 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.167 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.182 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.192 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.202 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.217 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.227 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.234 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.243 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21.251 UTC [11375] WARNING: there is no transaction in progress
2026-09-24 07:12:21,962 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:12:21,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:12:22,201 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:12:22,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:12:22,448 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:12:22,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:12:22,703 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:12:22,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:12:22,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:12:23,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cnv3.c.gcp-upstream.internal';"
2026-09-24 07:12:23,463 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:12:23,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:12:23,712 [I] DB_PATCHES have already been applied
I20260924 07:12:23.718065 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasks9UDai/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:12:23.718138 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 07:12:23.721629 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55475
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:12:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:12:24.726807 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.576s user 0.001s sys 0.009s
Sep 24, 2026 7:12:44 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:12:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:12:44 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:12:44 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:12:45 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:12:45.677937 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:12:45 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233965, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:12:45.785223 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:12:45.796047 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:12:46.835562 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:12:46.835655 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms
I20260924 07:12:46.835748 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:48263
2026-09-24 07:12:47,014 [I] DB FLAVOR :POSTGRES
2026-09-24 07:12:47,014 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:12:47,015 [I] Checking connection
2026-09-24 07:12:47,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:12:47,285 [I] connection success
2026-09-24 07:12:47,285 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:12:47,285 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:12:47,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:12:47,538 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:12:47,538 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:12:47,538 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:12:47,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:48263/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:12:47,794 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:12:48.291008 392 mini_ranger_kms.cc:310] Created kms service
I20260924 07:12:48.523489 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:12:48.567866 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:12:48 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233965, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:48.926505 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:12:48.926656 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:12:48.926676 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 07:12:48.930160 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11555
I20260924 07:12:48.930306 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34445
Sep 24, 2026 7:12:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:12:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:12:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:12:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:12:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:12:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:12:50.991246 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.206s user 0.025s sys 0.137s
I20260924 07:12:50.991410 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:12:50.991520 392 mini_ranger_kms.cc:401] 127.0.98.20:34445/kms/v1/keys
Sep 24 07:12:51 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:12:51 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:12:51 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233971, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:12:54 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:54.230801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36101
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34445/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:54.342448 11606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:54.342742 11606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:54.342800 11606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:54.346971 11606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:54.347051 11606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:54.347077 11606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:54.347108 11606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:54.347137 11606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:54.353042 11606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34445/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36101
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:54.354166 11606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:54.355229 11606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:54.364179 11611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:54.364144 11614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:54.364226 11606 server_base.cc:1034] running on GCE node
W20260924 07:12:54.364183 11612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:54.365090 11606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:54.366330 11606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:54.367466 11606 hybrid_clock.cc:648] HybridClock initialized: now 1790233974367406 us; error 76 us; skew 500 ppm
Sep 24 07:12:54 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:54.374429 11606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:54.375593 11606 webserver.cc:460] Webserver started at http://127.0.98.62:45123/ using document root <none> and password file <none>
I20260924 07:12:54.376206 11606 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:54.376266 11606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:54.376474 11606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:12:54 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:54.435350 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "16876de0f9b14124b8c6759e9e0e82d4"
format_stamp: "Formatted at 2026-09-24 07:12:54 on dist-test-slave-cnv3"
server_key: "b4e9fa8d8f5c16fbd6657ac6cb0dd0b9"
server_key_iv: "154cc50fb07c30e8f62911af599fbee7"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:54.436033 11606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "16876de0f9b14124b8c6759e9e0e82d4"
format_stamp: "Formatted at 2026-09-24 07:12:54 on dist-test-slave-cnv3"
server_key: "b4e9fa8d8f5c16fbd6657ac6cb0dd0b9"
server_key_iv: "154cc50fb07c30e8f62911af599fbee7"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:54.440058 11606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:12:54.455936 11621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:54.457365 11606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:54.457504 11606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "16876de0f9b14124b8c6759e9e0e82d4"
format_stamp: "Formatted at 2026-09-24 07:12:54 on dist-test-slave-cnv3"
server_key: "b4e9fa8d8f5c16fbd6657ac6cb0dd0b9"
server_key_iv: "154cc50fb07c30e8f62911af599fbee7"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:54.457634 11606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:54.473771 11606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:54.477073 11606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:54.477257 11606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:54.485466 11606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36101
I20260924 07:12:54.485507 11673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36101 every 8 connection(s)
I20260924 07:12:54.486474 11606 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:54.488459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 11606
I20260924 07:12:54.488569 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:54.488852 392 external_mini_cluster.cc:1428] Setting key 9ec3d0a7a5763cd1fc4f50ece127fa93
I20260924 07:12:54.489713 11674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:54.496382 11674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: Bootstrap starting.
I20260924 07:12:54.498997 11674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: Neither blocks nor log segments found. Creating new log.
Sep 24 07:12:54 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:54.499872 11674 log.cc:824] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:54.501894 11674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: No bootstrap required, opened a new log
I20260924 07:12:54.506141 11674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260924 07:12:54.506376 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:54.490345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.0.98.62:36101)
0924 07:12:54.490812 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:54.490832 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:54.490878 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:12:54.491830 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:54.491855 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:54.492300 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:54.492710 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:54.493578 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:54.495169 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:54.496373 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:54.497095 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:54.497245 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:54.500287 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:54.500294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:54.500312 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:54.500358 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:54.502965 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:54.503518 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:54.503525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:54.503531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:54.503659 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:54.503977 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:54.503980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:54.503982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:54.504612 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:54.504836 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:54.505113 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 07:12:54.505422 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":135,"threads_started":1}
I20260924 07:12:54.506425 11674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:54.506469 11674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16876de0f9b14124b8c6759e9e0e82d4, State: Initialized, Role: FOLLOWER
I20260924 07:12:54.506995 11674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260924 07:12:54.507148 11674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:54.507232 11674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:54.507329 11674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:54.508291 11674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260924 07:12:54.508628 11674 leader_election.cc:302] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16876de0f9b14124b8c6759e9e0e82d4; no voters:
I20260924 07:12:54.508963 11674 leader_election.cc:288] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:54.509092 11679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:54.509366 11679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [term 1 LEADER]: Becoming Leader. State: Replica: 16876de0f9b14124b8c6759e9e0e82d4, State: Running, Role: LEADER
I20260924 07:12:54.509660 11679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } }
I20260924 07:12:54.510228 11674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:54.510905 11679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16876de0f9b14124b8c6759e9e0e82d4. Latest consensus state: current_term: 1 leader_uuid: "16876de0f9b14124b8c6759e9e0e82d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260924 07:12:54.511099 11679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:54.511190 11680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16876de0f9b14124b8c6759e9e0e82d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16876de0f9b14124b8c6759e9e0e82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36101 } } }
I20260924 07:12:54.511361 11680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:54.511545 11688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:54.514410 11688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:54.514925 11674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:12:54.524798 11688 catalog_manager.cc:1347] Generated new cluster ID: 29eee8bf633b417eaee51a735dd5d8c5
I20260924 07:12:54.524945 11688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:54.534356 11688 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:54.535771 11688 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:54.544662 11688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: Generated new TSK 0
I20260924 07:12:54.545955 11688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:12:55.250540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:55.403455 11719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:55.403800 11719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:55.403896 11719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:55.410123 11719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:55.410272 11719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:55.410447 11719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:55.416545 11719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:55.417901 11719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:55.419117 11719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:55.428987 11726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:55.429039 11727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:55.429334 11719 server_base.cc:1034] running on GCE node
W20260924 07:12:55.429008 11729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:55.429951 11719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:55.430899 11719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:55.432104 11719 hybrid_clock.cc:648] HybridClock initialized: now 1790233975432083 us; error 42 us; skew 500 ppm
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:55.435935 11719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:55.437119 11719 webserver.cc:460] Webserver started at http://127.0.98.1:43477/ using document root <none> and password file <none>
I20260924 07:12:55.437832 11719 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:55.437927 11719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:55.438277 11719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:55.460258 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "30a8ea04012e42cfa549f2ae22234812"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "725a2d00062c2184b22e262d1fe9aa5f"
server_key_iv: "7844926d1239fc50828b68c3df78b7f3"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.461118 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30a8ea04012e42cfa549f2ae22234812"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "725a2d00062c2184b22e262d1fe9aa5f"
server_key_iv: "7844926d1239fc50828b68c3df78b7f3"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.466624 11719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:12:55.501698 11736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.503719 11719 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.004s sys 0.000s
I20260924 07:12:55.503896 11719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "30a8ea04012e42cfa549f2ae22234812"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "725a2d00062c2184b22e262d1fe9aa5f"
server_key_iv: "7844926d1239fc50828b68c3df78b7f3"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.504046 11719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:55.529220 11719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:55.532994 11719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:55.533258 11719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:55.534128 11719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:55.535243 11719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:55.535343 11719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.535516 11719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:55.535599 11719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.546427 11719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41909
I20260924 07:12:55.547813 11719 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:55.550582 11851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41909 every 8 connection(s)
I20260924 07:12:55.553311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 11719
I20260924 07:12:55.553434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:55.553702 392 external_mini_cluster.cc:1428] Setting key 5870072a2c060bae98040c0735c38075
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:12:55.566551 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:55.551010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56627 (local address 127.0.98.62:36101)
0924 07:12:55.551290 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:55.551295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:55.551844 (+ 549us) server_negotiation.cc:407] Connection header received
0924 07:12:55.553201 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:55.553205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:55.553273 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:55.553344 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:55.555129 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.556034 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.557953 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.558305 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.558413 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:55.561754 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:55.561759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:55.561762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:55.561798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:55.563684 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:55.564533 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:55.564537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:55.564539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:55.564641 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:55.565094 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:55.565098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:55.565100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:55.565354 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:55.565438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:55.566214 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 07:12:55.566364 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":66,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:12:55.567749 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:55.551052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.1:56627)
0924 07:12:55.551651 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:55.551720 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:55.552914 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:55.553836 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:55.553847 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:55.554217 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:55.554951 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:55.554971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.556244 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.556248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:55.557798 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:55.557810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.558656 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.558660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:55.558888 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:55.559730 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:55.559753 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:55.561496 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:55.563876 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:55.563887 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:55.563916 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:55.564400 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:55.564757 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:55.564760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:55.564762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:55.564980 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:55.565524 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:55.565531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:55.565911 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 07:12:55.566149 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 07:12:55.566553 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":151,"threads_started":1}
I20260924 07:12:55.569850 11852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260924 07:12:55.570137 11852 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:55.570791 11852 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260924 07:12:55.572795 11638 ts_manager.cc:194] Registered new tserver with Master: 30a8ea04012e42cfa549f2ae22234812 (127.0.98.1:41909)
I20260924 07:12:55.574419 11638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:12:55.586892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:12:55.741128 11858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:55.741561 11858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:55.741652 11858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:55.748231 11858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:55.748342 11858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:55.748433 11858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:55.757134 11858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:55.758613 11858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:55.760068 11858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:55.772807 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:55.772807 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:55.773250 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:55.776665 11858 server_base.cc:1034] running on GCE node
I20260924 07:12:55.777288 11858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:55.778117 11858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:55.779323 11858 hybrid_clock.cc:648] HybridClock initialized: now 1790233975779288 us; error 50 us; skew 500 ppm
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:55.783066 11858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:55.784348 11858 webserver.cc:460] Webserver started at http://127.0.98.2:33585/ using document root <none> and password file <none>
I20260924 07:12:55.785162 11858 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:55.785255 11858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:55.785529 11858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:55.804590 11858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe7045d45e0a4efeb2b81f31663df945"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "966d396b2112905710db611829b0f7da"
server_key_iv: "4f39cf254775f4c25dd6263293f1232d"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.805378 11858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe7045d45e0a4efeb2b81f31663df945"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "966d396b2112905710db611829b0f7da"
server_key_iv: "4f39cf254775f4c25dd6263293f1232d"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.811884 11858 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 07:12:55.827544 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.829182 11858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:12:55.829319 11858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "fe7045d45e0a4efeb2b81f31663df945"
format_stamp: "Formatted at 2026-09-24 07:12:55 on dist-test-slave-cnv3"
server_key: "966d396b2112905710db611829b0f7da"
server_key_iv: "4f39cf254775f4c25dd6263293f1232d"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:55.829468 11858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:55.848290 11858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:55.851483 11858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:55.851649 11858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:55.852252 11858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:55.853154 11858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:55.853202 11858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.853245 11858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:55.853273 11858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:55.870512 11858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33101
I20260924 07:12:55.870539 11987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33101 every 8 connection(s)
I20260924 07:12:55.871675 11858 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:55.876798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 11858
I20260924 07:12:55.876925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:55.877213 392 external_mini_cluster.cc:1428] Setting key bc4713410b38ba7d3af14b32039addf0
Sep 24 07:12:55 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:12:55.891938 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:55.876028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44801 (local address 127.0.98.62:36101)
0924 07:12:55.876157 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:55.876162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:55.878756 (+ 2594us) server_negotiation.cc:407] Connection header received
0924 07:12:55.880718 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:55.880722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:55.880784 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:55.880834 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:55.882528 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.883044 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.884657 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.885018 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.885096 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:55.888110 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:55.888114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:55.888117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:55.888148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:55.889735 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:55.890233 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:55.890236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:55.890238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:55.890324 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:55.890658 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:55.890661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:55.890664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:55.890836 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:55.890926 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:55.891559 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 07:12:55.891761 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:12:55.893411 11990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:55.874792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.2:44801)
0924 07:12:55.878480 (+ 3688us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:55.878548 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:55.880423 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:55.881301 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:55.881311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:55.881685 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:55.882375 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:55.882390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.883167 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.883170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:55.884515 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:55.884527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:55.885314 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:55.885317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:55.885531 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:55.886231 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:55.886252 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:55.887901 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:55.889849 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:55.889856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:55.889874 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:55.890120 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:55.890416 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:55.890419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:55.890421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:55.890564 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:55.890938 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:55.890945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:55.891261 (+ 316us) client_negotiation.cc:769] Sending connection context
0924 07:12:55.891506 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 07:12:55.891836 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3426,"thread_start_us":154,"threads_started":1}
I20260924 07:12:55.894996 11988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260924 07:12:55.895273 11988 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:55.895960 11988 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260924 07:12:55.897132 11638 ts_manager.cc:194] Registered new tserver with Master: fe7045d45e0a4efeb2b81f31663df945 (127.0.98.2:33101)
I20260924 07:12:55.897905 11638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:12:55.910764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34445/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:12:56 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 07:12:56.077121 11993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:56.077541 11993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:56.077610 11993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:56.083907 11993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:56.083997 11993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:56.084084 11993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:56.091851 11993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34445/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:12:56.093276 11993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:56.094585 11993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:56.107470 11999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:56.107465 12001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:56.107981 11993 server_base.cc:1034] running on GCE node
W20260924 07:12:56.107998 11998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:56.108649 11993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:56.109508 11993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:56.110780 11993 hybrid_clock.cc:648] HybridClock initialized: now 1790233976110738 us; error 60 us; skew 500 ppm
Sep 24 07:12:56 dist-test-slave-cnv3 krb5kdc[11162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:56.114378 11993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:56.115617 11993 webserver.cc:460] Webserver started at http://127.0.98.3:44077/ using document root <none> and password file <none>
I20260924 07:12:56.116333 11993 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:56.116430 11993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:56.116652 11993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:12:56 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:56.135072 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd63f4597dde4475bd2deb5cb1d56b7a"
format_stamp: "Formatted at 2026-09-24 07:12:56 on dist-test-slave-cnv3"
server_key: "783bd0baf78ab23ad54bf1769c91dec8"
server_key_iv: "fce5b7266c662d6b1b8a5bf251984d48"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:56.135871 11993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd63f4597dde4475bd2deb5cb1d56b7a"
format_stamp: "Formatted at 2026-09-24 07:12:56 on dist-test-slave-cnv3"
server_key: "783bd0baf78ab23ad54bf1769c91dec8"
server_key_iv: "fce5b7266c662d6b1b8a5bf251984d48"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:56.145931 11993 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260924 07:12:56.167802 12008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:56.169358 11993 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260924 07:12:56.169505 11993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "fd63f4597dde4475bd2deb5cb1d56b7a"
format_stamp: "Formatted at 2026-09-24 07:12:56 on dist-test-slave-cnv3"
server_key: "783bd0baf78ab23ad54bf1769c91dec8"
server_key_iv: "fce5b7266c662d6b1b8a5bf251984d48"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:56.169655 11993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:56.180974 11993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:56.183949 11993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:56.184122 11993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:56.184724 11993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:56.185601 11993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:56.185649 11993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:56.185700 11993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:56.185731 11993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:56.195657 11993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41717
I20260924 07:12:56.195688 12121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41717 every 8 connection(s)
I20260924 07:12:56.196624 11993 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:56.201880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 11993
I20260924 07:12:56.202015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:56.202308 392 external_mini_cluster.cc:1428] Setting key 5211fa90dda09810ff61db5cb6bbf4e2
Sep 24 07:12:56 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:56.211000 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.198537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60627 (local address 127.0.98.62:36101)
0924 07:12:56.198666 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.198671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.199653 (+ 982us) server_negotiation.cc:407] Connection header received
0924 07:12:56.200831 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.200835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.200897 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.200968 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:56.202270 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.202986 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.204701 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.204935 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.205022 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.207323 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:56.207326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:56.207328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:56.207353 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:56.208696 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:56.209250 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:56.209254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:56.209256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:56.209354 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:56.209704 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:56.209707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:56.209709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:56.209906 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:56.209996 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.210688 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.210830 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:12:56.212100 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.198989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36101 (local address 127.0.98.3:60627)
0924 07:12:56.199501 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.199555 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.200619 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.201152 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.201159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.201446 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:56.202096 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.202117 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.203171 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.203174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.204588 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.204600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.205076 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.205080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.205258 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.205825 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:56.205842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:56.207153 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:56.208810 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:56.208816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:56.208836 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:56.209106 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:56.209448 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:56.209452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:56.209454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:56.209616 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:56.210009 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:56.210017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:56.210427 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.210633 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.210998 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260924 07:12:56.213279 12122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36101
I20260924 07:12:56.213537 12122 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:56.214071 12122 heartbeater.cc:511] Master 127.0.98.62:36101 requested a full tablet report, sending...
I20260924 07:12:56.215219 11638 ts_manager.cc:194] Registered new tserver with Master: fd63f4597dde4475bd2deb5cb1d56b7a (127.0.98.3:41717)
I20260924 07:12:56.215981 11638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60627
I20260924 07:12:56.216253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:56.226593 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.218873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.98.62:36101)
0924 07:12:56.218994 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.218998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.219259 (+ 261us) server_negotiation.cc:407] Connection header received
0924 07:12:56.219401 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.219405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.219453 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.219519 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:56.220221 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.220744 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.221783 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.221999 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.222091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.223658 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:56.223663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:56.223666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:56.223692 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:56.225018 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:56.225436 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:56.225439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:56.225440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:56.225528 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:56.225774 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:56.225777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:56.225779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:56.225964 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:56.226079 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.226347 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.226468 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:12:56.229903 11638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:56.233363 11638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:56.253860 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.244939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41717 (local address 127.0.0.1:37984)
0924 07:12:56.246320 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.246332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.246407 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.247520 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.247527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.247552 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:56.247682 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.247689 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.250030 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.250034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.251309 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.251320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.253307 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.253311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.253425 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.253468 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.253568 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.253696 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1321,"thread_start_us":86,"threads_started":1}
I20260924 07:12:56.254572 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.245429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41909 (local address 127.0.0.1:41876)
0924 07:12:56.247591 (+ 2162us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.247606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.247709 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.250264 (+ 2555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.250268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.250289 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:56.250403 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.250409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.251903 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.251907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.252945 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.252955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.254203 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.254207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.254300 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.254320 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.254381 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.254448 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2097,"thread_start_us":94,"threads_started":1}
I20260924 07:12:56.254946 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.245111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.0.98.3:41717)
0924 07:12:56.246894 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.246905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.246926 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:12:56.246993 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.246997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.247170 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.247307 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:56.247844 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.249895 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.251529 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.253186 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.254505 (+ 1319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.254653 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.254712 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.254814 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622,"thread_start_us":130,"threads_started":1}
I20260924 07:12:56.255904 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.245814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.0.98.1:41909)
0924 07:12:56.246634 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.246641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.247805 (+ 1164us) server_negotiation.cc:407] Connection header received
0924 07:12:56.247888 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.247893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.248106 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.248262 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:56.250547 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.251771 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.253094 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.253880 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.253993 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.254144 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.254834 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.254925 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":114,"threads_started":1}
I20260924 07:12:56.258250 12056 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb (DEFAULT_TABLE table=test-table [id=8a66cac4696d4b0d96350d2b1c3d0f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:56.258672 11786 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb (DEFAULT_TABLE table=test-table [id=8a66cac4696d4b0d96350d2b1c3d0f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:56.259450 12056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:56.259783 11786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:56.268106 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.244179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33101 (local address 127.0.0.1:56740)
0924 07:12:56.245642 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.245701 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.245835 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.262877 (+ 17042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.262880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.262902 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:56.263024 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.263030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.264551 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.264555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.265817 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.265830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.267639 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.267643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.267749 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.267774 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.267845 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.267948 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"thread_start_us":109,"threads_started":1}
I20260924 07:12:56.268767 12140 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Bootstrap starting.
I20260924 07:12:56.268837 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.244339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.2:33101)
0924 07:12:56.262181 (+ 17842us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.262194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.262223 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:12:56.262311 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.262317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.262536 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.262710 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:56.263163 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.264408 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.266454 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.267292 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.267417 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.267572 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.268588 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.268697 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17710,"thread_start_us":95,"threads_started":1}
I20260924 07:12:56.271193 12140 tablet_bootstrap.cc:654] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:56.272106 12140 log.cc:824] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:56.272209 11922 tablet_service.cc:1467] Processing CreateTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb (DEFAULT_TABLE table=test-table [id=8a66cac4696d4b0d96350d2b1c3d0f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:56.273169 12141 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Bootstrap starting.
I20260924 07:12:56.274148 11922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:56.277349 12140 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: No bootstrap required, opened a new log
I20260924 07:12:56.277592 12140 ts_tablet_manager.cc:1397] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260924 07:12:56.282194 12140 raft_consensus.cc:348] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.282730 12140 raft_consensus.cc:374] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:56.282840 12140 raft_consensus.cc:729] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a8ea04012e42cfa549f2ae22234812, State: Initialized, Role: FOLLOWER
I20260924 07:12:56.284103 12140 consensus_queue.cc:260] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.284813 11852 heartbeater.cc:503] Master 127.0.98.62:36101 was elected leader, sending a full tablet report...
I20260924 07:12:56.286278 12140 ts_tablet_manager.cc:1428] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260924 07:12:56.286693 12141 tablet_bootstrap.cc:654] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:56.287896 12141 log.cc:824] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:56.289315 12143 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Bootstrap starting.
I20260924 07:12:56.292292 12143 tablet_bootstrap.cc:654] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:56.294042 12141 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: No bootstrap required, opened a new log
I20260924 07:12:56.294248 12141 ts_tablet_manager.cc:1397] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260924 07:12:56.298307 12141 raft_consensus.cc:348] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.298576 12141 raft_consensus.cc:374] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:56.298658 12141 raft_consensus.cc:729] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd63f4597dde4475bd2deb5cb1d56b7a, State: Initialized, Role: FOLLOWER
I20260924 07:12:56.299114 12141 consensus_queue.cc:260] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.300220 12141 ts_tablet_manager.cc:1428] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 07:12:56.300647 12122 heartbeater.cc:503] Master 127.0.98.62:36101 was elected leader, sending a full tablet report...
W20260924 07:12:56.300928 11853 tablet.cc:2369] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:56.303956 12143 log.cc:824] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:56.306434 12143 tablet_bootstrap.cc:492] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: No bootstrap required, opened a new log
I20260924 07:12:56.306650 12143 ts_tablet_manager.cc:1397] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260924 07:12:56.311295 12143 raft_consensus.cc:348] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.311815 12143 raft_consensus.cc:374] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:56.311939 12143 raft_consensus.cc:729] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7045d45e0a4efeb2b81f31663df945, State: Initialized, Role: FOLLOWER
I20260924 07:12:56.312516 12143 consensus_queue.cc:260] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.313627 12143 ts_tablet_manager.cc:1428] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:12:56.314277 11988 heartbeater.cc:503] Master 127.0.98.62:36101 was elected leader, sending a full tablet report...
Sep 24 07:12:56 dist-test-slave-cnv3 krb5kdc[11162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 07:12:56.381040 11989 tablet.cc:2369] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:56.386618 12144 raft_consensus.cc:482] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:56.386897 12144 raft_consensus.cc:504] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.388494 12144 leader_election.cc:288] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe7045d45e0a4efeb2b81f31663df945 (127.0.98.2:33101), fd63f4597dde4475bd2deb5cb1d56b7a (127.0.98.3:41717)
I20260924 07:12:56.394441 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.389551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60247 (local address 127.0.98.2:33101)
0924 07:12:56.389820 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.389824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.389838 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:56.389891 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.389894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.389940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.390011 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:56.390523 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.391280 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.393040 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.393672 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.393758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.393835 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.394203 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.394294 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260924 07:12:56.395114 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.388679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33101 (local address 127.0.98.1:60247)
0924 07:12:56.388991 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.389007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.389110 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.390125 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.390128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.390169 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:56.390358 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.390366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.391502 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.391507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.392880 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.392892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.393940 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.393944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.394051 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.394076 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.394854 (+ 778us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.394947 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":36,"spinlock_wait_cycles":5504,"thread_start_us":115,"threads_started":1}
I20260924 07:12:56.396036 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdf5a05a4b14709bd7db8b5335cf1cb" candidate_uuid: "30a8ea04012e42cfa549f2ae22234812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7045d45e0a4efeb2b81f31663df945" is_pre_election: true
I20260924 07:12:56.396409 11924 raft_consensus.cc:2454] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a8ea04012e42cfa549f2ae22234812 in term 0.
I20260924 07:12:56.397195 11740 leader_election.cc:302] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a8ea04012e42cfa549f2ae22234812, fe7045d45e0a4efeb2b81f31663df945; no voters:
I20260924 07:12:56.397574 12144 raft_consensus.cc:2781] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:56.397658 12144 raft_consensus.cc:482] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:56.397698 12144 raft_consensus.cc:3037] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:56.399063 12144 raft_consensus.cc:504] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.399487 12144 leader_election.cc:288] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [CANDIDATE]: Term 1 election: Requested vote from peers fe7045d45e0a4efeb2b81f31663df945 (127.0.98.2:33101), fd63f4597dde4475bd2deb5cb1d56b7a (127.0.98.3:41717)
I20260924 07:12:56.400058 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdf5a05a4b14709bd7db8b5335cf1cb" candidate_uuid: "30a8ea04012e42cfa549f2ae22234812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe7045d45e0a4efeb2b81f31663df945"
I20260924 07:12:56.400199 11924 raft_consensus.cc:3037] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:56.401558 11924 raft_consensus.cc:2454] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a8ea04012e42cfa549f2ae22234812 in term 1.
I20260924 07:12:56.401978 11740 leader_election.cc:302] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a8ea04012e42cfa549f2ae22234812, fe7045d45e0a4efeb2b81f31663df945; no voters:
I20260924 07:12:56.402199 12144 raft_consensus.cc:2781] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:56.402391 12144 raft_consensus.cc:686] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [term 1 LEADER]: Becoming Leader. State: Replica: 30a8ea04012e42cfa549f2ae22234812, State: Running, Role: LEADER
I20260924 07:12:56.402787 12144 consensus_queue.cc:237] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } }
I20260924 07:12:56.406999 11638 catalog_manager.cc:5495] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a8ea04012e42cfa549f2ae22234812 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "30a8ea04012e42cfa549f2ae22234812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a8ea04012e42cfa549f2ae22234812" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 } health_report { overall_health: UNKNOWN } } }
I20260924 07:12:56.428829 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.389686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46887 (local address 127.0.98.3:41717)
0924 07:12:56.389819 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.389824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.389838 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:56.389891 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.389894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.389940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.390010 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:56.396099 (+ 6089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.396859 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.427294 (+ 30435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.428018 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.428118 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.428199 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.428589 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.428681 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:12:56.429296 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.389092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41717 (local address 127.0.98.1:46887)
0924 07:12:56.389377 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.389389 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:56.389515 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:56.390125 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:56.390128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:56.390167 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:56.395923 (+ 5756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.395932 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.406062 (+ 10130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.406066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.427048 (+ 20982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:56.427074 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.428311 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.428315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:56.428435 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.428461 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:12:56.429027 (+ 566us) client_negotiation.cc:240] Negotiation successful
0924 07:12:56.429121 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"spinlock_wait_cycles":28288,"thread_start_us":98,"threads_started":1}
I20260924 07:12:56.430063 12075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdf5a05a4b14709bd7db8b5335cf1cb" candidate_uuid: "30a8ea04012e42cfa549f2ae22234812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" is_pre_election: true
I20260924 07:12:56.430449 12075 raft_consensus.cc:2454] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a8ea04012e42cfa549f2ae22234812 in term 0.
I20260924 07:12:56.430747 12076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cdf5a05a4b14709bd7db8b5335cf1cb" candidate_uuid: "30a8ea04012e42cfa549f2ae22234812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a"
I20260924 07:12:56.430878 12076 raft_consensus.cc:3037] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:56.432336 12076 raft_consensus.cc:2454] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a8ea04012e42cfa549f2ae22234812 in term 1.
W20260924 07:12:56.448069 12123 tablet.cc:2369] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:56.461862 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.456789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.0.98.1:41909)
0924 07:12:56.456906 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:56.456910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:56.456924 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:56.457035 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:56.457039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:56.457103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:56.457201 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:56.457582 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.458391 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.459678 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:56.460025 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:56.460115 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:56.460524 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:56.461111 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:56.461221 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:56.461650 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 07:12:56.461718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:12:56.479358 12076 raft_consensus.cc:1261] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 1 FOLLOWER]: Refusing update from remote peer 30a8ea04012e42cfa549f2ae22234812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:56.480386 12144 consensus_queue.cc:1035] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd63f4597dde4475bd2deb5cb1d56b7a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:56.494084 11924 raft_consensus.cc:1261] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 1 FOLLOWER]: Refusing update from remote peer 30a8ea04012e42cfa549f2ae22234812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:56.494830 12144 consensus_queue.cc:1035] T 0cdf5a05a4b14709bd7db8b5335cf1cb P 30a8ea04012e42cfa549f2ae22234812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7045d45e0a4efeb2b81f31663df945" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:56.501685 12156 mvcc.cc:205] Tried to move back new op lower bound from 7332798367623778304 to 7332798367350185984. Current Snapshot: MvccSnapshot[applied={T|T < 7332798367623778304}]
I20260924 07:12:56.512267 12159 mvcc.cc:205] Tried to move back new op lower bound from 7332798367623778304 to 7332798367350185984. Current Snapshot: MvccSnapshot[applied={T|T < 7332798367623778304}]
I20260924 07:12:56.522256 11638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:56.522465 11638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:56.525373 11638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16876de0f9b14124b8c6759e9e0e82d4: Sending DeleteTablet for 3 replicas of tablet 0cdf5a05a4b14709bd7db8b5335cf1cb
I20260924 07:12:56.526780 11922 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56740
I20260924 07:12:56.527410 12056 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37984
I20260924 07:12:56.528604 11786 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41876
I20260924 07:12:56.528906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 11719
I20260924 07:12:56.531589 12166 tablet_replica.cc:330] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: stopping tablet replica
I20260924 07:12:56.531989 12166 raft_consensus.cc:2229] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:56.532387 12166 raft_consensus.cc:2258] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:56.532799 12165 tablet_replica.cc:330] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: stopping tablet replica
I20260924 07:12:56.533126 12165 raft_consensus.cc:2229] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:56.533402 12165 raft_consensus.cc:2258] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:56.538298 12165 ts_tablet_manager.cc:1905] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:56.539141 12166 ts_tablet_manager.cc:1905] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:56.545755 12165 ts_tablet_manager.cc:1918] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:56.545893 12165 log.cc:1196] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/0cdf5a05a4b14709bd7db8b5335cf1cb
I20260924 07:12:56.546222 12165 ts_tablet_manager.cc:1939] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fe7045d45e0a4efeb2b81f31663df945: Deleting consensus metadata
I20260924 07:12:56.550218 12166 ts_tablet_manager.cc:1918] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:56.550350 12166 log.cc:1196] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/0cdf5a05a4b14709bd7db8b5335cf1cb
I20260924 07:12:56.550684 12166 ts_tablet_manager.cc:1939] T 0cdf5a05a4b14709bd7db8b5335cf1cb P fd63f4597dde4475bd2deb5cb1d56b7a: Deleting consensus metadata
I20260924 07:12:56.552238 11624 catalog_manager.cc:4841] TS fd63f4597dde4475bd2deb5cb1d56b7a (127.0.98.3:41717): tablet 0cdf5a05a4b14709bd7db8b5335cf1cb (table test-table [id=8a66cac4696d4b0d96350d2b1c3d0f12]) successfully deleted
W20260924 07:12:56.553143 11625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41909 (error 108)
I20260924 07:12:56.554214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 11858
I20260924 07:12:56.556221 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.555659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41909 (local address 127.0.0.1:41894)
0924 07:12:56.556050 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.556128 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
W20260924 07:12:56.562017 11625 catalog_manager.cc:4584] TS 30a8ea04012e42cfa549f2ae22234812 (127.0.98.1:41909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0cdf5a05a4b14709bd7db8b5335cf1cb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41909: connect: Connection refused (error 111)
I20260924 07:12:56.562877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 11993
I20260924 07:12:56.564177 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:56.563674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33101 (local address 127.0.0.1:56750)
0924 07:12:56.564039 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:56.564092 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
I20260924 07:12:56.570848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 11606
I20260924 07:12:56.578302 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:12:56.923906 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:12:56.924048 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:12:57.259333 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:12:57.259 UTC [11192] LOG: received smart shutdown request
2026-09-24 07:12:57.263 UTC [11192] LOG: background worker "logical replication launcher" (PID 11201) exited with exit code 1
2026-09-24 07:12:57.263 UTC [11196] LOG: shutting down
2026-09-24 07:12:57.440 UTC [11192] LOG: database system is shut down
2026-09-24T07:12:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46010 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:57 dist-test-slave-cnv3 krb5kdc[12177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:57 dist-test-slave-cnv3 krb5kdc[12177](info): set up 2 sockets
Sep 24 07:12:57 dist-test-slave-cnv3 krb5kdc[12177](info): commencing operation
krb5kdc: starting...
W20260924 07:12:58.484004 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:58 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 07:12:58.663260 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasks9UDai/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/postgres -l logfile start
2026-09-24 07:13:01.372 UTC [12207] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:13:01.372 UTC [12207] LOG: listening on IPv4 address "127.0.98.20", port 53677
2026-09-24 07:13:01.376 UTC [12207] LOG: listening on Unix socket "/tmp/.s.PGSQL.53677"
2026-09-24 07:13:01.396 UTC [12210] LOG: database system was shut down at 2026-09-24 07:12:59 UTC
2026-09-24 07:13:01.400 UTC [12207] LOG: database system is ready to accept connections
I20260924 07:13:02.271149 392 mini_postgres.cc:96] Postgres bound to 53677
2026-09-24 07:13:02.277 UTC [12218] FATAL: database "slave" does not exist
127.0.98.20:53677 - accepting connections
I20260924 07:13:02.277882 392 mini_ranger.cc:161] Starting Ranger...
I20260924 07:13:02.292637 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:13:02.813581 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:13:02.813651 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin
2026-09-24 07:13:03,079 [I] DB FLAVOR :POSTGRES
2026-09-24 07:13:03,080 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:13:03,080 [I] Checking connection..
2026-09-24 07:13:03,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:13:03,345 [I] Checking connection passed.
2026-09-24 07:13:03,345 [I] --------- Verifying version history table ---------
2026-09-24 07:13:03,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:13:03,594 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:13:03,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:13:03,861 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:13:03,861 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:13:03,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:13:04.125 UTC [12282] WARNING: there is no transaction in progress
2026-09-24 07:13:04,131 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:13:04,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:13:04,402 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:13:04,402 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:13:04,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:13:04,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:13:04,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cnv3.c.gcp-upstream.internal','N') ;"
2026-09-24 07:13:05,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:13:05,417 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:13:05,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:13:05,682 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:13:05,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:13:05,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:13:06,187 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:13:06,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:13:07.229 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.248 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.267 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.285 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.302 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.322 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.391 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.400 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.416 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.427 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.441 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.456 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.467 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.474 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.481 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:07.488 UTC [12390] WARNING: there is no transaction in progress
2026-09-24 07:13:08,243 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:13:08,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:13:08,499 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:13:08,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:13:08,780 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:13:08,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:13:09,033 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:13:09,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:13:09,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:13:09,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cnv3.c.gcp-upstream.internal';"
2026-09-24 07:13:09,748 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:13:09,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:13:10,000 [I] DB_PATCHES have already been applied
I20260924 07:13:10.006524 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasks9UDai/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:13:10.006590 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 07:13:10.009824 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:58317
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:13:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:13:11.021203 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.743s user 0.000s sys 0.009s
Sep 24, 2026 7:13:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:13:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:13:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:13:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:13:33.521848 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:13:33 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234013, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:13:33.628589 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:13:33.638944 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:13:34.614054 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:13:34.614125 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms
I20260924 07:13:34.614143 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53677
2026-09-24 07:13:34,770 [I] DB FLAVOR :POSTGRES
2026-09-24 07:13:34,770 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:13:34,770 [I] Checking connection
2026-09-24 07:13:34,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:13:35,008 [I] connection success
2026-09-24 07:13:35,008 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:13:35,008 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:13:35,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:13:35,265 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:13:35,265 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:13:35,265 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:13:35,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53677/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:13:35,534 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:13:35.971153 392 mini_ranger_kms.cc:310] Created kms service
I20260924 07:13:36.171819 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:13:36.216243 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:13:36 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234013, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:36.630162 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:13:36.630442 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms:/tmp/dist-test-tasks9UDai/build/debug/bin/postgresql.jar:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasks9UDai/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:13:36.630488 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 07:13:36.634125 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12570
I20260924 07:13:36.634208 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34785
Sep 24, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:13:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:13:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:13:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:13:38.658910 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.030s user 0.000s sys 0.140s
I20260924 07:13:38.659065 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:13:38.659144 392 mini_ranger_kms.cc:401] 127.0.98.20:34785/kms/v1/keys
Sep 24 07:13:38 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:13:39 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:13:39 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:13:41 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:41.972675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34729
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:42.098150 12621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:42.098464 12621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:42.098520 12621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:42.102733 12621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:42.102825 12621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:42.102855 12621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:42.102875 12621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:42.102896 12621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:42.109228 12621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34785/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34729
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:42.110400 12621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:42.111510 12621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:42.121814 12626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:42.121806 12627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:42.121874 12621 server_base.cc:1034] running on GCE node
W20260924 07:13:42.122196 12629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:42.122834 12621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:42.124150 12621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:42.125355 12621 hybrid_clock.cc:648] HybridClock initialized: now 1790234022125338 us; error 31 us; skew 500 ppm
Sep 24 07:13:42 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:42.134377 12621 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:42.135573 12621 webserver.cc:460] Webserver started at http://127.0.98.62:36711/ using document root <none> and password file <none>
I20260924 07:13:42.136204 12621 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:42.136268 12621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:42.136502 12621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:13:42 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:42.199322 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "41187415a75d49d4a8286db93a62813b"
format_stamp: "Formatted at 2026-09-24 07:13:42 on dist-test-slave-cnv3"
server_key: "9bae5036611481309fb38d358359e3a5"
server_key_iv: "195983a201ec7e0811f54a53676c8515"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:42.200415 12621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "41187415a75d49d4a8286db93a62813b"
format_stamp: "Formatted at 2026-09-24 07:13:42 on dist-test-slave-cnv3"
server_key: "9bae5036611481309fb38d358359e3a5"
server_key_iv: "195983a201ec7e0811f54a53676c8515"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:42.204990 12621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:13:42.220252 12636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:42.221693 12621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:13:42.221819 12621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "41187415a75d49d4a8286db93a62813b"
format_stamp: "Formatted at 2026-09-24 07:13:42 on dist-test-slave-cnv3"
server_key: "9bae5036611481309fb38d358359e3a5"
server_key_iv: "195983a201ec7e0811f54a53676c8515"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:42.221975 12621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:42.244937 12621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:42.248327 12621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:42.248530 12621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:42.256345 12621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34729
I20260924 07:13:42.256361 12688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34729 every 8 connection(s)
I20260924 07:13:42.257323 12621 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:42.260186 12689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:42.261694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 12621
I20260924 07:13:42.261816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:42.262097 392 external_mini_cluster.cc:1428] Setting key b1847a1c4b3eab1ab599a71fa973c98f
I20260924 07:13:42.266873 12689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Bootstrap starting.
I20260924 07:13:42.269418 12689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:42.270318 12689 log.cc:824] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:13:42 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:42.272686 12689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: No bootstrap required, opened a new log
I20260924 07:13:42.276114 12689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:42.276412 12689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:42.276489 12689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41187415a75d49d4a8286db93a62813b, State: Initialized, Role: FOLLOWER
I20260924 07:13:42.276978 12689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:42.277132 12689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:42.277215 12689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:42.277309 12689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:42.278455 12689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:42.278848 12689 leader_election.cc:302] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41187415a75d49d4a8286db93a62813b; no voters:
I20260924 07:13:42.279172 12689 leader_election.cc:288] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:42.279323 12694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:42.279515 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:42.263653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.0.98.62:34729)
0924 07:13:42.264112 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:42.264131 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:42.264175 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:13:42.264996 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:42.265020 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:42.265465 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:42.265859 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:42.266732 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:42.268174 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:42.269412 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:42.270095 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:42.270260 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:42.273175 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:42.273182 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:42.273200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:42.273232 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:42.275825 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:42.276349 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:42.276356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:42.276362 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:42.276511 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:42.276830 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:42.276833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:42.276835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:42.277360 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:42.277555 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:42.278020 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 07:13:42.278336 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":142,"threads_started":1}
I20260924 07:13:42.279587 12694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 LEADER]: Becoming Leader. State: Replica: 41187415a75d49d4a8286db93a62813b, State: Running, Role: LEADER
I20260924 07:13:42.280122 12694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:42.280539 12689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:42.281855 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41187415a75d49d4a8286db93a62813b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260924 07:13:42.282065 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:42.282399 12695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41187415a75d49d4a8286db93a62813b. Latest consensus state: current_term: 1 leader_uuid: "41187415a75d49d4a8286db93a62813b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260924 07:13:42.282536 12695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:42.282652 12689 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:13:42.282874 12708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:42.291141 12708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:42.303934 12708 catalog_manager.cc:1347] Generated new cluster ID: 391c51571d364c5b9388f71e12b98e6a
I20260924 07:13:42.304015 12708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:42.332796 12708 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:42.333879 12708 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:42.341377 12708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Generated new TSK 0
I20260924 07:13:42.342136 12708 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:13:43.019438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:43.158663 12734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:43.159096 12734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:43.159168 12734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:43.164113 12734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:43.164273 12734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:43.164424 12734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:43.171569 12734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:43.172739 12734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:43.173787 12734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:43.184029 12742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:43.184293 12744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.184376 12734 server_base.cc:1034] running on GCE node
W20260924 07:13:43.184665 12741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.185140 12734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:43.185853 12734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:43.187022 12734 hybrid_clock.cc:648] HybridClock initialized: now 1790234023186992 us; error 47 us; skew 500 ppm
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:43.190240 12734 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:43.191450 12734 webserver.cc:460] Webserver started at http://127.0.98.1:39995/ using document root <none> and password file <none>
I20260924 07:13:43.192186 12734 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:43.192324 12734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:43.192588 12734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:43.215469 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "94a28b9892dc4a368de3607593816927"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "68ca447bff8d12bbca022a98f6f89f8b"
server_key_iv: "d97ce918655432fed8fffcfccaa3ac4a"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.216339 12734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94a28b9892dc4a368de3607593816927"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "68ca447bff8d12bbca022a98f6f89f8b"
server_key_iv: "d97ce918655432fed8fffcfccaa3ac4a"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.221884 12734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 07:13:43.236598 12751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.238353 12734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:13:43.238574 12734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "94a28b9892dc4a368de3607593816927"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "68ca447bff8d12bbca022a98f6f89f8b"
server_key_iv: "d97ce918655432fed8fffcfccaa3ac4a"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.238715 12734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:43.265903 12734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:43.269552 12734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:43.269814 12734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:43.270588 12734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:43.271873 12734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:43.271998 12734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.272131 12734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:43.272223 12734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.286414 12734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36849
I20260924 07:13:43.287794 12734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:43.292850 12865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36849 every 8 connection(s)
I20260924 07:13:43.293936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 12734
I20260924 07:13:43.294075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:43.294363 392 external_mini_cluster.cc:1428] Setting key 42e06e51d5a73891e02800b2dcd2b5a1
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:13:43.305598 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.290713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.1:41779)
0924 07:13:43.291332 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:43.291435 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:43.292501 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:43.293173 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:43.293183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:43.293548 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:43.294213 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.294232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.295177 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.295180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.296271 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.296281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.297021 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.297025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.297225 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.297822 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:43.297843 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:43.299601 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:43.301912 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.301920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.301942 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.302342 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.302692 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.302695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.302697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.302903 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.303443 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:43.303462 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:43.303849 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 07:13:43.304069 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 07:13:43.304438 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":164,"threads_started":1}
I20260924 07:13:43.306442 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.291915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41779 (local address 127.0.98.62:34729)
0924 07:13:43.292189 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:43.292197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:43.292215 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:13:43.292779 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:43.292782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:43.292845 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:43.292948 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:43.294393 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.295049 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.296403 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.296734 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.296837 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.299824 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:43.299827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:43.299830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:43.299860 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:43.301755 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.302482 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.302486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.302488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.302589 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.303013 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.303016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.303018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.303260 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:43.303349 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:43.306156 (+ 2807us) server_negotiation.cc:299] Negotiation successful
0924 07:13:43.306277 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
I20260924 07:13:43.307487 12867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:43.307744 12867 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:13:43.308346 12867 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:43.311352 12653 ts_manager.cc:194] Registered new tserver with Master: 94a28b9892dc4a368de3607593816927 (127.0.98.1:36849)
I20260924 07:13:43.313005 12653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:13:43.334580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:13:43.493286 12873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:43.493785 12873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:43.493907 12873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:43.500286 12873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:43.500376 12873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:43.500470 12873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:43.508734 12873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:43.510403 12873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:43.511829 12873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:43.524972 12880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.525264 12873 server_base.cc:1034] running on GCE node
W20260924 07:13:43.525475 12882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:43.526147 12879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.526906 12873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:43.527815 12873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:43.529006 12873 hybrid_clock.cc:648] HybridClock initialized: now 1790234023528970 us; error 53 us; skew 500 ppm
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:43.532851 12873 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:43.534067 12873 webserver.cc:460] Webserver started at http://127.0.98.2:37793/ using document root <none> and password file <none>
I20260924 07:13:43.534852 12873 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:43.535019 12873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:43.535311 12873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:43.552225 12873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c95430bb03043bcbe84df37d0b38b7f"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "c1373cc8b2375b932ba41e10b9e9113a"
server_key_iv: "394c1826fd601e6156277ff846437c86"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.552949 12873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c95430bb03043bcbe84df37d0b38b7f"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "c1373cc8b2375b932ba41e10b9e9113a"
server_key_iv: "394c1826fd601e6156277ff846437c86"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.561049 12873 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260924 07:13:43.582970 12889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.584964 12873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:13:43.585137 12873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "1c95430bb03043bcbe84df37d0b38b7f"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "c1373cc8b2375b932ba41e10b9e9113a"
server_key_iv: "394c1826fd601e6156277ff846437c86"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.585304 12873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:43.600531 12873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:43.603943 12873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:43.604135 12873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:43.604725 12873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:43.605612 12873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:43.605662 12873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.605715 12873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:43.605744 12873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.615586 12873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38953
I20260924 07:13:43.615674 13002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38953 every 8 connection(s)
I20260924 07:13:43.616714 12873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:43.617792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 12873
I20260924 07:13:43.617908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:43.618230 392 external_mini_cluster.cc:1428] Setting key eb1d16e2981d71b9018e343a93c33b10
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:13:43.631964 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.619058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48193 (local address 127.0.98.62:34729)
0924 07:13:43.619179 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:43.619183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:43.620403 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 07:13:43.621661 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:43.621665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:43.621719 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:43.621790 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:43.623036 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.623828 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.625198 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.625438 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.625554 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.628199 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:43.628203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:43.628206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:43.628235 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:43.629574 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.630190 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.630193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.630194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.630277 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.630667 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.630671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.630672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.630865 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:43.630994 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:43.631667 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 07:13:43.631798 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:13:43.632998 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.619688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.2:48193)
0924 07:13:43.620245 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:43.620301 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:43.621412 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:43.621983 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:43.621990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:43.622253 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:43.622854 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.622873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.623966 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.623970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.625062 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.625071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.625576 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.625580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.625778 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.626475 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:43.626497 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:43.628034 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:43.629685 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.629691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.629709 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.630071 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.630370 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.630374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.630375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.630568 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.631002 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:43.631009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:43.631358 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 07:13:43.631603 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 07:13:43.631936 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":146,"threads_started":1}
I20260924 07:13:43.634241 13003 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:43.634482 13003 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:43.634984 13003 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:43.636018 12653 ts_manager.cc:194] Registered new tserver with Master: 1c95430bb03043bcbe84df37d0b38b7f (127.0.98.2:38953)
I20260924 07:13:43.636567 12653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:13:43.658788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 07:13:43.825666 13008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:43.826083 13008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:43.826289 13008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:43.832710 13008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:43.832868 13008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:43.833007 13008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:43.840791 13008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:43.842468 13008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:43.843981 13008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:43.858175 13016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:43.858175 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.859921 13008 server_base.cc:1034] running on GCE node
W20260924 07:13:43.860236 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.860833 13008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:43.861675 13008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:43.864951 13008 hybrid_clock.cc:648] HybridClock initialized: now 1790234023864849 us; error 106 us; skew 500 ppm
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:43.868989 13008 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:43.870265 13008 webserver.cc:460] Webserver started at http://127.0.98.3:39653/ using document root <none> and password file <none>
I20260924 07:13:43.871032 13008 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:43.871162 13008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:43.871472 13008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:43.886637 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "f637b6c3888f4ae09d23d65058a1fc19"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "b0ee2cb0877eac3bad002582a5225a6d"
server_key_iv: "1abe8d00660287c5400f35525f860594"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.887235 13008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f637b6c3888f4ae09d23d65058a1fc19"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "b0ee2cb0877eac3bad002582a5225a6d"
server_key_iv: "1abe8d00660287c5400f35525f860594"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.892342 13008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 07:13:43.912544 13023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.914021 13008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:43.914155 13008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "f637b6c3888f4ae09d23d65058a1fc19"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "b0ee2cb0877eac3bad002582a5225a6d"
server_key_iv: "1abe8d00660287c5400f35525f860594"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:43.914286 13008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:43.930361 13008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:43.933336 13008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:43.933504 13008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:43.934099 13008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:43.934919 13008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:43.934969 13008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.935024 13008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:43.935058 13008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.944482 13008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32799
I20260924 07:13:43.944528 13136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32799 every 8 connection(s)
I20260924 07:13:43.945513 13008 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:43.949988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13008
I20260924 07:13:43.950109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:43.950366 392 external_mini_cluster.cc:1428] Setting key 9ac4069aad548611872a0fa88f087047
Sep 24 07:13:43 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:43.959995 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.947511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38627 (local address 127.0.98.62:34729)
0924 07:13:43.947644 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:43.947649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:43.948605 (+ 956us) server_negotiation.cc:407] Connection header received
0924 07:13:43.949696 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:43.949699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:43.949760 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:43.949826 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:43.951988 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.952729 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.954053 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.954279 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.954365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.956661 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:43.956665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:43.956667 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:43.956695 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:43.957987 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.958468 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.958471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.958472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.958543 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:43.958862 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:43.958864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:43.958866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:43.959057 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:43.959148 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:43.959714 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 07:13:43.959832 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:13:43.960897 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.947976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.3:38627)
0924 07:13:43.948448 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:43.948502 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:43.949446 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:43.950848 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:43.950856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:43.951122 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:43.951826 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.951848 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.952882 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.952885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.953928 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:43.953938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.954424 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.954428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:43.954614 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.955178 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:43.955194 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:43.956499 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:43.958096 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.958102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.958119 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.958368 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.958620 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:43.958623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:43.958625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:43.958770 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:43.959154 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:43.959161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:43.959476 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 07:13:43.959654 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 07:13:43.959973 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":126,"threads_started":1}
I20260924 07:13:43.961951 13137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:43.962181 13137 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:43.962670 13137 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:43.963896 12653 ts_manager.cc:194] Registered new tserver with Master: f637b6c3888f4ae09d23d65058a1fc19 (127.0.98.3:32799)
I20260924 07:13:43.964588 12653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38627
I20260924 07:13:43.964905 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:43.965711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 12734
I20260924 07:13:43.971843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 12873
I20260924 07:13:43.978183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13008
I20260924 07:13:43.984469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 12621
I20260924 07:13:43.989678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--webserver_interface=127.0.98.62
--webserver_port=36711
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34729
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:44.100679 13140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:44.100968 13140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:44.101015 13140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:44.105086 13140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:44.105163 13140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:44.105189 13140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:44.105212 13140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:44.105230 13140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:44.110882 13140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34729
--ranger_config_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:44.112145 13140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:44.113200 13140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:44.121636 13145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.121610 13148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.121704 13140 server_base.cc:1034] running on GCE node
W20260924 07:13:44.121601 13146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.122453 13140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:44.123567 13140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:44.124737 13140 hybrid_clock.cc:648] HybridClock initialized: now 1790234024124708 us; error 41 us; skew 500 ppm
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:44.127543 13140 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:44.128496 13140 webserver.cc:460] Webserver started at http://127.0.98.62:36711/ using document root <none> and password file <none>
I20260924 07:13:44.128997 13140 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:44.129060 13140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:44.174293 13140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:44.176788 13156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.178159 13140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:13:44.178287 13140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "41187415a75d49d4a8286db93a62813b"
format_stamp: "Formatted at 2026-09-24 07:13:42 on dist-test-slave-cnv3"
server_key: "9bae5036611481309fb38d358359e3a5"
server_key_iv: "195983a201ec7e0811f54a53676c8515"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:44.178853 13140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:44.216087 13140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:44.219120 13140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:44.219313 13140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:44.226433 13140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34729
I20260924 07:13:44.226452 13208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34729 every 8 connection(s)
I20260924 07:13:44.227722 13140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:44.232568 13209 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:13:44.234316 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Bootstrap starting.
I20260924 07:13:44.237252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13140
I20260924 07:13:44.237792 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36849
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34729
--webserver_port=39995
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:44.251523 13209 log.cc:824] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:44.258397 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:44.258786 13209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Bootstrap complete.
I20260924 07:13:44.262770 13209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:44.263151 13209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41187415a75d49d4a8286db93a62813b, State: Initialized, Role: FOLLOWER
I20260924 07:13:44.263692 13209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:44.263811 13209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:44.263872 13209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:44.263960 13209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:13:44.265843 13209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:44.266152 13209 leader_election.cc:302] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41187415a75d49d4a8286db93a62813b; no voters:
I20260924 07:13:44.266464 13209 leader_election.cc:288] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:13:44.266590 13213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:13:44.266948 13213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [term 2 LEADER]: Becoming Leader. State: Replica: 41187415a75d49d4a8286db93a62813b, State: Running, Role: LEADER
I20260924 07:13:44.267397 13213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } }
I20260924 07:13:44.267688 13209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:44.268826 13214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41187415a75d49d4a8286db93a62813b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260924 07:13:44.268949 13214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:44.269618 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41187415a75d49d4a8286db93a62813b. Latest consensus state: current_term: 2 leader_uuid: "41187415a75d49d4a8286db93a62813b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41187415a75d49d4a8286db93a62813b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34729 } } }
I20260924 07:13:44.269708 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:44.269949 13209 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:13:44.270067 13222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:44.275887 13222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:44.276594 13222 catalog_manager.cc:1259] Loaded cluster ID: 391c51571d364c5b9388f71e12b98e6a
I20260924 07:13:44.276696 13222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:44.279359 13222 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:44.280148 13222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Loaded TSK: 0
I20260924 07:13:44.281718 13222 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:13:44.383059 13211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:44.383518 13211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:44.383625 13211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:44.390144 13211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:44.390307 13211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:44.390427 13211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:44.398237 13211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39995
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:44.399785 13211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:44.401134 13211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:44.413956 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.414019 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.413962 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.414204 13211 server_base.cc:1034] running on GCE node
I20260924 07:13:44.414996 13211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:44.415853 13211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:44.417042 13211 hybrid_clock.cc:648] HybridClock initialized: now 1790234024417012 us; error 47 us; skew 500 ppm
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:44.420882 13211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:44.422176 13211 webserver.cc:460] Webserver started at http://127.0.98.1:39995/ using document root <none> and password file <none>
I20260924 07:13:44.422928 13211 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:44.423105 13211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:44.454457 13211 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.003s sys 0.000s
I20260924 07:13:44.458212 13258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.460228 13211 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:13:44.460381 13211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "94a28b9892dc4a368de3607593816927"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "68ca447bff8d12bbca022a98f6f89f8b"
server_key_iv: "d97ce918655432fed8fffcfccaa3ac4a"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:44.461120 13211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:44.487231 13211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:44.490283 13211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:44.490479 13211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:44.491051 13211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:44.491963 13211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:44.492013 13211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.492067 13211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:44.492120 13211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.502142 13211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36849
I20260924 07:13:44.502328 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36849 every 8 connection(s)
I20260924 07:13:44.503557 13211 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:44.505550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13211
I20260924 07:13:44.506012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38953
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34729
--webserver_port=37793
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:44.528147 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.508409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.1:58169)
0924 07:13:44.509967 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:44.510030 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:44.510981 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:44.512952 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:44.512962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:44.513327 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:44.514026 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:44.514047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.516037 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.516041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:44.517491 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:44.517504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.518933 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.518938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:44.519167 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:44.519914 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:44.519938 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:44.521785 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:44.524334 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:44.524340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:44.524356 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:44.524687 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:44.525094 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:44.525097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:44.525099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:44.525288 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:44.526099 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:44.526106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:44.526451 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 07:13:44.526644 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 07:13:44.526998 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":122,"threads_started":1}
I20260924 07:13:44.530292 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.508260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58169 (local address 127.0.98.62:34729)
0924 07:13:44.511582 (+ 3322us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:44.511616 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:44.511665 (+ 49us) server_negotiation.cc:407] Connection header received
0924 07:13:44.511977 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:44.511992 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:44.512426 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:44.512813 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:44.514252 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.515857 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.517658 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.518462 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.518681 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:44.522029 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:44.522036 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:44.522054 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:44.522087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:44.524180 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:44.524830 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:44.524836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:44.524843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:44.524985 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:44.525396 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:44.525399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:44.525401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:44.525894 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:44.526034 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:44.528369 (+ 2335us) server_negotiation.cc:299] Negotiation successful
0924 07:13:44.528741 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3134,"thread_start_us":143,"threads_started":1}
I20260924 07:13:44.533074 13372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:44.533337 13372 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.533977 13372 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
W20260924 07:13:44.537860 13372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34729 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:13:44.538040 13372 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.538287 13372 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:44.539444 13372 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.539667 13372 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
W20260924 07:13:44.540643 13372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:13:44.642274 13374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:44.642552 13374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:44.642594 13374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:44.646879 13374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:44.646962 13374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:44.647047 13374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:44.653303 13374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37793
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:44.654457 13374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:44.655570 13374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:44.665839 13382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.665839 13384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.666260 13381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.666765 13374 server_base.cc:1034] running on GCE node
I20260924 07:13:44.667258 13374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:44.668009 13374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:44.669281 13374 hybrid_clock.cc:648] HybridClock initialized: now 1790234024669238 us; error 66 us; skew 500 ppm
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:44.671973 13374 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:44.673061 13374 webserver.cc:460] Webserver started at http://127.0.98.2:37793/ using document root <none> and password file <none>
I20260924 07:13:44.673815 13374 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:44.673928 13374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:44.714741 13374 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.000s sys 0.003s
I20260924 07:13:44.720045 13392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.721988 13374 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260924 07:13:44.722155 13374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "1c95430bb03043bcbe84df37d0b38b7f"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "c1373cc8b2375b932ba41e10b9e9113a"
server_key_iv: "394c1826fd601e6156277ff846437c86"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:44.722985 13374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:44.750455 13374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:44.754561 13374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:44.754778 13374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:44.755563 13374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:44.756778 13374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:44.756891 13374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.756991 13374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:44.757056 13374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.768860 13374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38953
I20260924 07:13:44.770884 13374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:44.771819 13506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38953 every 8 connection(s)
I20260924 07:13:44.778041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13374
I20260924 07:13:44.778527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34785/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32799
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34729
--webserver_port=39653
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:44.787156 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.773046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50881 (local address 127.0.98.62:34729)
0924 07:13:44.773173 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:44.773176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:44.774302 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 07:13:44.775314 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:44.775318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:44.775419 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:44.775499 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:44.776731 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.777560 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.779297 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.779694 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.779813 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:44.782767 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:44.782772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:44.782775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:44.782811 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:44.784549 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:44.785222 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:44.785226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:44.785228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:44.785343 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:44.785762 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:44.785766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:44.785767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:44.786017 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:44.786115 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:44.786829 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 07:13:44.786978 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:13:44.788321 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.773606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.2:50881)
0924 07:13:44.774139 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:44.774184 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:44.775042 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:44.775713 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:44.775721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:44.776004 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:44.776568 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:44.776585 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.777712 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.777716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:44.779147 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:44.779160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:44.779903 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:44.779908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:44.780138 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:44.780805 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:44.780827 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:44.782540 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:44.784700 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:44.784706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:44.784727 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:44.785103 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:44.785459 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:44.785462 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:44.785464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:44.785665 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:44.786154 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:44.786161 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:44.786531 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 07:13:44.786770 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 07:13:44.787136 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":158,"threads_started":1}
I20260924 07:13:44.789685 13507 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:44.789949 13507 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.790573 13507 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
W20260924 07:13:44.792407 13507 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34729 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:13:44.792526 13507 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.792686 13507 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:44.793493 13507 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:44.793643 13507 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
W20260924 07:13:44.796713 13507 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:13:44.916962 13510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:44.917460 13510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:44.917590 13510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:44.923625 13510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:44.923687 13510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:44.923753 13510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:44.929855 13510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34785/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39653
--tserver_master_addrs=127.0.98.62:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:44.930977 13510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:44.932158 13510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:44.943010 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.943249 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.943074 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.944110 13510 server_base.cc:1034] running on GCE node
I20260924 07:13:44.944634 13510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:44.945511 13510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:44.946689 13510 hybrid_clock.cc:648] HybridClock initialized: now 1790234024946644 us; error 60 us; skew 500 ppm
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:44.949608 13510 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:44.950567 13510 webserver.cc:460] Webserver started at http://127.0.98.3:39653/ using document root <none> and password file <none>
I20260924 07:13:44.951128 13510 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:44.951211 13510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:13:44 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:44.977044 13510 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.003s
I20260924 07:13:44.980747 13526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.982610 13510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:13:44.982834 13510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "f637b6c3888f4ae09d23d65058a1fc19"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-cnv3"
server_key: "b0ee2cb0877eac3bad002582a5225a6d"
server_key_iv: "1abe8d00660287c5400f35525f860594"
server_key_version: "kuduclusterkey@0"
I20260924 07:13:44.983776 13510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:44.999318 13510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:45.003176 13510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:45.003400 13510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:45.004128 13510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:45.005188 13510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:45.005254 13510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:45.005311 13510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:45.005373 13510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:45.016667 13510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32799
I20260924 07:13:45.016717 13639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32799 every 8 connection(s)
I20260924 07:13:45.018227 13510 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:45.020648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13510
Sep 24 07:13:45 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:45.042143 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.022973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44021 (local address 127.0.98.62:34729)
0924 07:13:45.023098 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.023103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.023335 (+ 232us) server_negotiation.cc:407] Connection header received
0924 07:13:45.028822 (+ 5487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.028826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.028898 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.028995 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:45.031030 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.031890 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.034013 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.034336 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.034419 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.037525 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:45.037529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:45.037532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:45.037562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:45.039403 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.040135 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.040139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.040141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.040242 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.040700 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.040703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.040705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.040968 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:45.041045 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.041817 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.041970 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:13:45.044993 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.021749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34729 (local address 127.0.98.3:44021)
0924 07:13:45.023161 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:45.023223 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:45.028496 (+ 5273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:45.029590 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:45.029603 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:45.030065 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:45.030838 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.030861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.032060 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.032063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.033578 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.033590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.034637 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.034641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.034866 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.035620 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:45.035643 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:45.037299 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:45.039563 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:45.039569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:45.039589 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:45.040010 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:45.040371 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:45.040374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:45.040376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:45.040583 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:45.041130 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:45.041137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:45.041516 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 07:13:45.042340 (+ 824us) client_negotiation.cc:240] Negotiation successful
0924 07:13:45.042716 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":157,"threads_started":1}
I20260924 07:13:45.046506 13640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34729
I20260924 07:13:45.046780 13640 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:45.047533 13640 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:45.049189 13173 ts_manager.cc:194] Registered new tserver with Master: f637b6c3888f4ae09d23d65058a1fc19 (127.0.98.3:32799)
I20260924 07:13:45.050804 13173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44021
I20260924 07:13:45.054682 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.023624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.0.98.62:34729)
0924 07:13:45.023850 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.023854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.033694 (+ 9840us) server_negotiation.cc:407] Connection header received
0924 07:13:45.033736 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.033739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.033812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.033894 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:45.043053 (+ 9159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.043945 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.045680 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.046017 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.046118 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.048753 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:45.048757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:45.048760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:45.048785 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:45.050555 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.051088 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.051092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.051094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.051183 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.051530 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.051533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.051535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.051744 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:45.051920 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.054304 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.054456 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
Sep 24 07:13:45 dist-test-slave-cnv3 krb5kdc[12177](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:45.540887 13372 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:45.541188 13372 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:45.542163 13173 ts_manager.cc:194] Registered new tserver with Master: 94a28b9892dc4a368de3607593816927 (127.0.98.1:36849)
I20260924 07:13:45.542928 13173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58169
I20260924 07:13:45.796962 13507 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:45.797282 13507 heartbeater.cc:511] Master 127.0.98.62:34729 requested a full tablet report, sending...
I20260924 07:13:45.798197 13173 ts_manager.cc:194] Registered new tserver with Master: 1c95430bb03043bcbe84df37d0b38b7f (127.0.98.2:38953)
I20260924 07:13:45.798966 13173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50881
I20260924 07:13:45.803828 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:45.819887 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.805370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.0.98.62:34729)
0924 07:13:45.809227 (+ 3857us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.809232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.809686 (+ 454us) server_negotiation.cc:407] Connection header received
0924 07:13:45.809843 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.809846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.809906 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.810009 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:45.810969 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.811848 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.813147 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.813509 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.813618 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.815616 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:45.815628 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:45.815631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:45.815666 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:45.817502 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.817945 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.817949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.817951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.818053 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.818373 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.818376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.818378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.818625 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:45.818755 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.819555 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.819696 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3755,"thread_start_us":92,"threads_started":1}
Sep 24 07:13:45 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:45.841509 13173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:45.847084 13173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:45.875255 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.862454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38953 (local address 127.0.0.1:40960)
0924 07:13:45.867069 (+ 4615us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:45.867081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:45.867184 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:45.868323 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:45.868329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:45.868350 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:45.868513 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.868519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.871224 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.871228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.873235 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.873251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.874667 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.874671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.874807 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.874847 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:13:45.874957 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:13:45.875090 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3538,"thread_start_us":115,"threads_started":1}
I20260924 07:13:45.877188 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.863759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.0.98.2:38953)
0924 07:13:45.867514 (+ 3755us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.867522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.867547 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:13:45.867627 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.867633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.867879 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.868028 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:45.868636 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.871088 (+ 2452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.873419 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.874319 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.874438 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.874602 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.876904 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.877039 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3603,"thread_start_us":99,"threads_started":1}
I20260924 07:13:45.877938 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.862385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32799 (local address 127.0.0.1:34338)
0924 07:13:45.866848 (+ 4463us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:45.866920 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:45.868197 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:45.873952 (+ 5755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:45.873956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:45.873982 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:45.874094 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.874100 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.875452 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.875456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.876517 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.876527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.877529 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.877533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.877620 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.877643 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:13:45.877731 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:13:45.877800 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3526,"thread_start_us":121,"threads_started":1}
I20260924 07:13:45.878763 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.863286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.0.98.3:32799)
0924 07:13:45.873146 (+ 9860us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.873154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.873186 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:13:45.873277 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.873283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.873576 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.873761 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:45.874231 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.875294 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.876649 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.877412 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.877514 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.877666 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.878517 (+ 851us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.878627 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9704,"thread_start_us":116,"threads_started":1}
I20260924 07:13:45.881309 13427 tablet_service.cc:1467] Processing CreateTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe (DEFAULT_TABLE table=test-table [id=203ddbe78dff4a33b030af0e74e7e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:45.882664 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce947b53a45e4af8a9bbea75ffaa33fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:45.882902 13566 tablet_service.cc:1467] Processing CreateTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe (DEFAULT_TABLE table=test-table [id=203ddbe78dff4a33b030af0e74e7e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:45.884122 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce947b53a45e4af8a9bbea75ffaa33fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:45.893465 13673 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Bootstrap starting.
I20260924 07:13:45.894279 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.865208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36849 (local address 127.0.0.1:51682)
0924 07:13:45.875980 (+ 10772us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:45.875994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:45.876060 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:45.878755 (+ 2695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:45.878758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:45.878777 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:45.878865 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.878871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.880201 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.880204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.889376 (+ 9172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:45.889399 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.893816 (+ 4417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.893821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:45.893935 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.893967 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:13:45.894035 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:13:45.894131 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10709,"thread_start_us":97,"threads_started":1}
I20260924 07:13:45.894644 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.865391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.0.98.1:36849)
0924 07:13:45.877925 (+ 12534us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.877932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.877959 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:13:45.878171 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.878178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.878440 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.878592 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:45.878994 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.880086 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.889553 (+ 9467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.891262 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.891424 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.894316 (+ 2892us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.894384 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.894501 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12399,"thread_start_us":104,"threads_started":1}
I20260924 07:13:45.896242 13674 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Bootstrap starting.
I20260924 07:13:45.896371 13673 tablet_bootstrap.cc:654] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:45.897316 13673 log.cc:824] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:45.897933 13305 tablet_service.cc:1467] Processing CreateTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe (DEFAULT_TABLE table=test-table [id=203ddbe78dff4a33b030af0e74e7e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:45.898721 13674 tablet_bootstrap.cc:654] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:45.899091 13305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce947b53a45e4af8a9bbea75ffaa33fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:45.899706 13674 log.cc:824] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:45.899953 13673 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: No bootstrap required, opened a new log
I20260924 07:13:45.900163 13673 ts_tablet_manager.cc:1397] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:13:45.902058 13674 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: No bootstrap required, opened a new log
I20260924 07:13:45.902261 13674 ts_tablet_manager.cc:1397] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:13:45.904536 13673 raft_consensus.cc:348] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.904812 13673 raft_consensus.cc:374] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:45.904870 13673 raft_consensus.cc:729] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c95430bb03043bcbe84df37d0b38b7f, State: Initialized, Role: FOLLOWER
I20260924 07:13:45.905404 13673 consensus_queue.cc:260] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.906510 13673 ts_tablet_manager.cc:1428] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:13:45.906642 13674 raft_consensus.cc:348] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.906874 13507 heartbeater.cc:503] Master 127.0.98.62:34729 was elected leader, sending a full tablet report...
I20260924 07:13:45.907691 13674 raft_consensus.cc:374] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:45.907817 13674 raft_consensus.cc:729] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f637b6c3888f4ae09d23d65058a1fc19, State: Initialized, Role: FOLLOWER
I20260924 07:13:45.908350 13674 consensus_queue.cc:260] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.909495 13674 ts_tablet_manager.cc:1428] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:13:45.909880 13640 heartbeater.cc:503] Master 127.0.98.62:34729 was elected leader, sending a full tablet report...
I20260924 07:13:45.912192 13679 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Bootstrap starting.
I20260924 07:13:45.915175 13679 tablet_bootstrap.cc:654] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:45.916180 13679 log.cc:824] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:45.918797 13679 tablet_bootstrap.cc:492] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: No bootstrap required, opened a new log
I20260924 07:13:45.919013 13679 ts_tablet_manager.cc:1397] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260924 07:13:45.922580 13679 raft_consensus.cc:348] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.922839 13679 raft_consensus.cc:374] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:45.922911 13679 raft_consensus.cc:729] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a28b9892dc4a368de3607593816927, State: Initialized, Role: FOLLOWER
I20260924 07:13:45.923396 13679 consensus_queue.cc:260] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:45.924147 13372 heartbeater.cc:503] Master 127.0.98.62:34729 was elected leader, sending a full tablet report...
I20260924 07:13:45.924561 13679 ts_tablet_manager.cc:1428] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260924 07:13:46.007364 13373 tablet.cc:2369] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:46.021832 13641 tablet.cc:2369] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:46.024331 13508 tablet.cc:2369] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:46.045969 13678 raft_consensus.cc:482] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:46.046237 13678 raft_consensus.cc:504] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:46.047762 13678 leader_election.cc:288] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c95430bb03043bcbe84df37d0b38b7f (127.0.98.2:38953), 94a28b9892dc4a368de3607593816927 (127.0.98.1:36849)
I20260924 07:13:46.053588 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.048473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51397 (local address 127.0.98.1:36849)
0924 07:13:46.048594 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:46.048598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:46.049017 (+ 419us) server_negotiation.cc:407] Connection header received
0924 07:13:46.049208 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:46.049212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:46.049261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:46.049308 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:46.049790 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.050570 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.052110 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.052756 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.052843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:46.052908 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:46.053260 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 07:13:46.053441 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:13:46.054013 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.048097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36849 (local address 127.0.98.3:51397)
0924 07:13:46.048920 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:46.048935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:46.049094 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:46.049465 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:46.049468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:46.049486 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:46.049588 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:46.049595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.050724 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.050727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:46.051954 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:46.051966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.053009 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.053012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:46.053107 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:46.053134 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:13:46.053762 (+ 628us) client_negotiation.cc:240] Negotiation successful
0924 07:13:46.053850 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260924 07:13:46.054775 13326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce947b53a45e4af8a9bbea75ffaa33fe" candidate_uuid: "f637b6c3888f4ae09d23d65058a1fc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a28b9892dc4a368de3607593816927" is_pre_election: true
I20260924 07:13:46.054905 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.048540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54917 (local address 127.0.98.2:38953)
0924 07:13:46.048671 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:46.048675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:46.048917 (+ 242us) server_negotiation.cc:407] Connection header received
0924 07:13:46.049125 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:46.049129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:46.049178 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:46.049222 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:46.049590 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.050320 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.051955 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.052539 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.052614 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:46.052685 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:46.054685 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0924 07:13:46.054782 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:13:46.055114 13326 raft_consensus.cc:2454] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f637b6c3888f4ae09d23d65058a1fc19 in term 0.
I20260924 07:13:46.055343 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.048032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38953 (local address 127.0.98.3:54917)
0924 07:13:46.048818 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:46.048833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:46.049008 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:46.049355 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:46.049357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:46.049384 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:46.049480 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:46.049486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.050475 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.050479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:46.051812 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:46.051822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.054407 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.054410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:46.054520 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:46.054544 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:13:46.055104 (+ 560us) client_negotiation.cc:240] Negotiation successful
0924 07:13:46.055188 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":127,"threads_started":1}
I20260924 07:13:46.055919 13529 leader_election.cc:302] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94a28b9892dc4a368de3607593816927, f637b6c3888f4ae09d23d65058a1fc19; no voters:
I20260924 07:13:46.056030 13457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce947b53a45e4af8a9bbea75ffaa33fe" candidate_uuid: "f637b6c3888f4ae09d23d65058a1fc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c95430bb03043bcbe84df37d0b38b7f" is_pre_election: true
I20260924 07:13:46.056258 13678 raft_consensus.cc:2781] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:46.056324 13457 raft_consensus.cc:2454] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f637b6c3888f4ae09d23d65058a1fc19 in term 0.
I20260924 07:13:46.056345 13678 raft_consensus.cc:482] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:46.056403 13678 raft_consensus.cc:3037] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:46.057708 13678 raft_consensus.cc:504] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:46.058079 13678 leader_election.cc:288] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [CANDIDATE]: Term 1 election: Requested vote from peers 1c95430bb03043bcbe84df37d0b38b7f (127.0.98.2:38953), 94a28b9892dc4a368de3607593816927 (127.0.98.1:36849)
I20260924 07:13:46.058465 13457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce947b53a45e4af8a9bbea75ffaa33fe" candidate_uuid: "f637b6c3888f4ae09d23d65058a1fc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c95430bb03043bcbe84df37d0b38b7f"
I20260924 07:13:46.058600 13457 raft_consensus.cc:3037] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:46.058638 13326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce947b53a45e4af8a9bbea75ffaa33fe" candidate_uuid: "f637b6c3888f4ae09d23d65058a1fc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94a28b9892dc4a368de3607593816927"
I20260924 07:13:46.058794 13326 raft_consensus.cc:3037] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:46.059934 13457 raft_consensus.cc:2454] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f637b6c3888f4ae09d23d65058a1fc19 in term 1.
I20260924 07:13:46.060221 13326 raft_consensus.cc:2454] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f637b6c3888f4ae09d23d65058a1fc19 in term 1.
I20260924 07:13:46.060315 13530 leader_election.cc:302] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c95430bb03043bcbe84df37d0b38b7f, f637b6c3888f4ae09d23d65058a1fc19; no voters:
I20260924 07:13:46.060482 13678 raft_consensus.cc:2781] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:46.060679 13678 raft_consensus.cc:686] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 1 LEADER]: Becoming Leader. State: Replica: f637b6c3888f4ae09d23d65058a1fc19, State: Running, Role: LEADER
I20260924 07:13:46.061062 13678 consensus_queue.cc:237] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } }
I20260924 07:13:46.064769 13173 catalog_manager.cc:5495] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 reported cstate change: term changed from 0 to 1, leader changed from <none> to f637b6c3888f4ae09d23d65058a1fc19 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f637b6c3888f4ae09d23d65058a1fc19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f637b6c3888f4ae09d23d65058a1fc19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:46.111222 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.105728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.98.3:32799)
0924 07:13:46.105845 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:46.105850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:46.106060 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:13:46.106160 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:46.106163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:46.106239 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:46.106351 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:46.106811 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.107634 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.108860 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:46.109194 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:46.109300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:46.109726 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:46.110349 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:46.110516 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:46.110975 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 07:13:46.111047 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:13:46.120357 13457 raft_consensus.cc:1261] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 1 FOLLOWER]: Refusing update from remote peer f637b6c3888f4ae09d23d65058a1fc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:46.120359 13326 raft_consensus.cc:1261] T ce947b53a45e4af8a9bbea75ffaa33fe P 94a28b9892dc4a368de3607593816927 [term 1 FOLLOWER]: Refusing update from remote peer f637b6c3888f4ae09d23d65058a1fc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:46.121206 13678 consensus_queue.cc:1035] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94a28b9892dc4a368de3607593816927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:46.121703 13678 consensus_queue.cc:1035] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c95430bb03043bcbe84df37d0b38b7f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:46.134270 13687 mvcc.cc:205] Tried to move back new op lower bound from 7332798570978123776 to 7332798570749685760. Current Snapshot: MvccSnapshot[applied={T|T < 7332798570978123776}]
I20260924 07:13:46.134332 13688 mvcc.cc:205] Tried to move back new op lower bound from 7332798570978123776 to 7332798570749685760. Current Snapshot: MvccSnapshot[applied={T|T < 7332798570978123776}]
I20260924 07:13:46.142285 13689 mvcc.cc:205] Tried to move back new op lower bound from 7332798570978123776 to 7332798570749685760. Current Snapshot: MvccSnapshot[applied={T|T < 7332798570978123776 or (T in {7332798570978123776})}]
I20260924 07:13:46.148875 13173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:46.149237 13173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:46.152601 13173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41187415a75d49d4a8286db93a62813b: Sending DeleteTablet for 3 replicas of tablet ce947b53a45e4af8a9bbea75ffaa33fe
I20260924 07:13:46.153779 13305 tablet_service.cc:1514] Processing DeleteTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51682
I20260924 07:13:46.154241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13211
I20260924 07:13:46.154385 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40960
I20260924 07:13:46.156250 13698 tablet_replica.cc:330] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f: stopping tablet replica
I20260924 07:13:46.156661 13698 raft_consensus.cc:2229] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:46.157063 13698 raft_consensus.cc:2258] T ce947b53a45e4af8a9bbea75ffaa33fe P 1c95430bb03043bcbe84df37d0b38b7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:46.157035 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet ce947b53a45e4af8a9bbea75ffaa33fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34338
Sep 24 07:13:46 dist-test-slave-cnv3 krb5kdc[12177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:13:46.163013 13699 tablet_replica.cc:330] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: stopping tablet replica
I20260924 07:13:46.168668 13699 raft_consensus.cc:2229] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:46.169235 13699 raft_consensus.cc:2258] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:46.171283 13699 ts_tablet_manager.cc:1905] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:46.174383 13699 ts_tablet_manager.cc:1918] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:46.174489 13699 log.cc:1196] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/ce947b53a45e4af8a9bbea75ffaa33fe
I20260924 07:13:46.174800 13699 ts_tablet_manager.cc:1939] T ce947b53a45e4af8a9bbea75ffaa33fe P f637b6c3888f4ae09d23d65058a1fc19: Deleting consensus metadata
W20260924 07:13:46.175660 13159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36849 (error 108)
I20260924 07:13:46.181757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13374
I20260924 07:13:46.188912 13158 catalog_manager.cc:4841] TS f637b6c3888f4ae09d23d65058a1fc19 (127.0.98.3:32799): tablet ce947b53a45e4af8a9bbea75ffaa33fe (table test-table [id=203ddbe78dff4a33b030af0e74e7e924]) successfully deleted
I20260924 07:13:46.189992 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.189642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36849 (local address 127.0.0.1:51694)
0924 07:13:46.189798 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:46.189895 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:13:46.190200 13159 catalog_manager.cc:4584] TS 94a28b9892dc4a368de3607593816927 (127.0.98.1:36849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce947b53a45e4af8a9bbea75ffaa33fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36849: connect: Connection refused (error 111)
I20260924 07:13:46.193130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13510
I20260924 07:13:46.196286 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:46.195920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38953 (local address 127.0.0.1:40962)
0924 07:13:46.196134 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:46.196190 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 07:13:46.205674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13140
I20260924 07:13:46.213737 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:13:46.559119 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:13:46.559242 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:13:46.894729 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:13:46.894 UTC [12207] LOG: received smart shutdown request
2026-09-24 07:13:46.899 UTC [12207] LOG: background worker "logical replication launcher" (PID 12216) exited with exit code 1
2026-09-24 07:13:46.899 UTC [12211] LOG: shutting down
2026-09-24 07:13:47.107 UTC [12207] LOG: database system is shut down
2026-09-24T07:13:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49680 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:47 dist-test-slave-cnv3 krb5kdc[13710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:47 dist-test-slave-cnv3 krb5kdc[13710](info): set up 2 sockets
Sep 24 07:13:47 dist-test-slave-cnv3 krb5kdc[13710](info): commencing operation
krb5kdc: starting...
W20260924 07:13:48.179312 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:48.306558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46105
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:48.416939 13723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:48.417248 13723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:48.417303 13723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:48.421504 13723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:48.421584 13723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:48.421619 13723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:48.421641 13723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:48.421667 13723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:48.427161 13723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:48.428308 13723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:48.429464 13723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:48.437855 13731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.438023 13723 server_base.cc:1034] running on GCE node
W20260924 07:13:48.437875 13729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:48.437897 13728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.438740 13723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:48.439967 13723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:48.441172 13723 hybrid_clock.cc:648] HybridClock initialized: now 1790234028441156 us; error 27 us; skew 500 ppm
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:48.443990 13723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:48.444882 13723 webserver.cc:460] Webserver started at http://127.0.98.62:46701/ using document root <none> and password file <none>
I20260924 07:13:48.445425 13723 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:48.445489 13723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:48.445690 13723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:48.447793 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "2dd418d0c4004026bedb7869f176e0eb"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "e56c2ec85a2f21a2764b41ca94b6152b"
server_key_iv: "9dd79d1383bded08d61c2b106153e810"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.448328 13723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "2dd418d0c4004026bedb7869f176e0eb"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "e56c2ec85a2f21a2764b41ca94b6152b"
server_key_iv: "9dd79d1383bded08d61c2b106153e810"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.451995 13723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:48.454486 13738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:48.455819 13723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:13:48.455935 13723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "2dd418d0c4004026bedb7869f176e0eb"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "e56c2ec85a2f21a2764b41ca94b6152b"
server_key_iv: "9dd79d1383bded08d61c2b106153e810"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.456039 13723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:48.475240 13723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:48.478509 13723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:48.478695 13723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:48.485420 13723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46105
I20260924 07:13:48.485468 13790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46105 every 8 connection(s)
I20260924 07:13:48.486388 13723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:48.489140 13791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:48.493008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13723
I20260924 07:13:48.493117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:48.492909 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.493404 392 external_mini_cluster.cc:1428] Setting key cf4604e270050b885c616be0be9c3f01
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:48.509336 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.495562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.0.98.62:46105)
0924 07:13:48.496092 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.496110 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.496219 (+ 109us) server_negotiation.cc:407] Connection header received
0924 07:13:48.497271 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.497275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.497447 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.497615 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.498458 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.499332 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.500321 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.500714 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.500860 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.504293 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.504300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.504314 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.504341 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.506105 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.506677 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.506683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.506684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.506789 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.507189 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.507192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.507193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.507675 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.507794 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.508236 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.508484 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":405,"spinlock_wait_cycles":22784,"thread_start_us":149,"threads_started":1}
I20260924 07:13:48.509338 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.495562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46105 (local address 127.0.0.1:44826)
0924 07:13:48.496029 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.496110 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.496962 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.497616 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.497621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.498062 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.498317 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.498330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.499457 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.499460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.500207 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.500216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.500855 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.500859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.501014 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.501572 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.501591 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.504125 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.506234 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.506243 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.506255 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.506529 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.506892 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.506896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.506898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.507066 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.507801 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.507809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.507983 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.508195 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.508555 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":63,"thread_start_us":113,"threads_started":1}
I20260924 07:13:48.511896 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.512610 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.512359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32768)
0924 07:13:48.512471 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.512539 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:13:48.512796 13741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
I20260924 07:13:48.513528 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.513337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32770)
0924 07:13:48.513421 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.513468 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:13:48.513661 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:13:48.520696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34469
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:48.562913 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 1
I20260924 07:13:48.563792 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.563457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32778)
0924 07:13:48.563592 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.563715 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:13:48.564697 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.564302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32784)
0924 07:13:48.564485 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.564560 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 07:13:48.564837 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
W20260924 07:13:48.632442 13798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:48.632750 13798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:48.632792 13798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:13:48.633014 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 2
I20260924 07:13:48.633940 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.633583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32786)
0924 07:13:48.633770 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.633844 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 07:13:48.634719 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.634516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32800)
0924 07:13:48.634611 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.634659 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:13:48.634842 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34469: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34469: connect: Connection refused (error 111)
W20260924 07:13:48.637542 13798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:48.637604 13798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:48.637640 13798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:48.637672 13798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:48.637701 13798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:48.643275 13798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:48.644493 13798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:48.645578 13798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:48.654175 13803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.654227 13798 server_base.cc:1034] running on GCE node
W20260924 07:13:48.654173 13806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:48.654166 13804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.654960 13798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:48.656227 13798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:48.657401 13798 hybrid_clock.cc:648] HybridClock initialized: now 1790234028657381 us; error 34 us; skew 500 ppm
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:48.665449 13798 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:13:48.666548 13798 webserver.cc:460] Webserver started at http://127.0.98.61:34775/ using document root <none> and password file <none>
I20260924 07:13:48.667140 13798 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:48.667215 13798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:48.667486 13798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:48.669720 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data/instance:
uuid: "6b3a1124211f44d9b867bf1a279d3119"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "5bdfce5b5e03f143afd637964a2deba7"
server_key_iv: "2d6624df6b4d9f53612635f4d1b7b059"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.670312 13798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal/instance:
uuid: "6b3a1124211f44d9b867bf1a279d3119"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "5bdfce5b5e03f143afd637964a2deba7"
server_key_iv: "2d6624df6b4d9f53612635f4d1b7b059"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.674330 13798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:13:48.676982 13813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:48.678424 13798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:13:48.678565 13798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "6b3a1124211f44d9b867bf1a279d3119"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "5bdfce5b5e03f143afd637964a2deba7"
server_key_iv: "2d6624df6b4d9f53612635f4d1b7b059"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.678682 13798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:48.713112 13798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:48.716755 13798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:48.716958 13798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:48.725073 13798 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34469
I20260924 07:13:48.725162 13865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34469 every 8 connection(s)
I20260924 07:13:48.725999 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } attempt: 3
I20260924 07:13:48.726480 13798 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:13:48.727530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13798
I20260924 07:13:48.727667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-1/wal/instance
I20260924 07:13:48.728075 392 external_mini_cluster.cc:1428] Setting key 71f5e4717429db6985fc1dbc6007c18d
I20260924 07:13:48.730428 13866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:48.736136 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 07:13:48.742992 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.726543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32814)
0924 07:13:48.726753 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.726776 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.726874 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.731128 (+ 4254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.731131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.731336 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.731525 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.731534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.733349 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.733354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.734197 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.734212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.735280 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.735284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.735428 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.735935 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.735952 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.737503 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.740359 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.740363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.740365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.740625 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.741065 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.741068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.741070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.741169 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.742472 (+ 1303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.742475 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.742582 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.742677 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.742817 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260924 07:13:48.744912 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.728882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.98.61:34469)
0924 07:13:48.729307 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.729326 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.729372 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:13:48.730245 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.730281 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.730748 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.731165 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.731706 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.733160 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.734381 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.735125 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.735308 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.737836 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.737844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.737862 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.737898 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.740241 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.740780 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.740787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.740793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.740952 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.741784 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.741787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.741789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.742349 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.742480 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.743490 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.743828 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:48.747486 13791 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.748260 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.747996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35640)
0924 07:13:48.748116 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.748174 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:13:48.749647 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.749335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35656)
0924 07:13:48.749525 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.749574 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260924 07:13:48.749811 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.751535 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.738811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46105 (local address 127.0.0.1:44828)
0924 07:13:48.741431 (+ 2620us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.741489 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.741612 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.742035 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.742040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.742363 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.742512 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.742520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.743552 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.743555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.744526 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.744537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.745246 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.745250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.745356 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.745952 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.745965 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.747578 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.749646 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.749658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.749664 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.749980 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.750271 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.750274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.750276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.750361 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.750750 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.750758 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.750986 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.751117 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.751326 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2411,"thread_start_us":99,"threads_started":1}
I20260924 07:13:48.751596 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.739144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.98.62:46105)
0924 07:13:48.739273 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.739277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.741569 (+ 2292us) server_negotiation.cc:407] Connection header received
0924 07:13:48.741737 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.741740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.741805 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.741891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.742644 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.743407 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.744660 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.745024 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.745101 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.747722 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.747725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.747728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.747750 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.749526 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.750080 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.750084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.750085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.750165 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.750448 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.750452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.750454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.750646 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.750728 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.751291 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.751434 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:13:48.752686 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:13:48.762001 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.753205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32830)
0924 07:13:48.753323 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.753336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.753414 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.753622 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.753625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.753852 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.753978 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.753987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.754943 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.754947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.755758 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.755779 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.756224 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.756228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.756327 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.756788 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.756802 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.758371 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.760552 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.760555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.760558 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.760748 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.761040 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.761043 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.761045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.761148 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.761501 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.761504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.761588 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.761678 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.761825 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:13:48.762328 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.753344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.0.98.61:34469)
0924 07:13:48.753444 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.753449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.753462 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:13:48.753490 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.753493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.753528 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.753604 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.754097 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.754857 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.755875 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.756115 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.756211 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.758522 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.758526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.758529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.758554 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.760463 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.760835 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.760839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.760841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.760914 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.761226 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.761229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.761231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.761421 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.761512 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.761806 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.761943 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:13:48.762614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44705
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:48.763106 13866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.763967 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.763649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35664)
0924 07:13:48.763790 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.763894 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 07:13:48.764428 13815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.765357 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.765141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35674)
0924 07:13:48.765248 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.765296 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:13:48.765522 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.780973 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 1
I20260924 07:13:48.781786 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.781500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35690)
0924 07:13:48.781637 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.781700 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:13:48.782550 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.782320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35692)
0924 07:13:48.782411 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.782490 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:13:48.782667 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.814836 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 1
I20260924 07:13:48.815788 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.815449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35706)
0924 07:13:48.815610 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.815684 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:13:48.816532 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.816315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35722)
0924 07:13:48.816413 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.816467 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:13:48.816717 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.857839 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 2
I20260924 07:13:48.858646 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.858358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35724)
0924 07:13:48.858503 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.858567 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 07:13:48.859534 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.859288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35730)
0924 07:13:48.859425 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.859475 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 07:13:48.859681 13791 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
W20260924 07:13:48.875667 13872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:48.875954 13872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:48.875995 13872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:48.880067 13872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:48.880136 13872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:48.880156 13872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:48.880172 13872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:48.880185 13872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:48.884877 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 2
I20260924 07:13:48.885711 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.885390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35734)
0924 07:13:48.885551 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.885622 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:13:48.885838 13872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46105,127.0.98.61:34469,127.0.98.60:44705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:48.886456 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.886229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35738)
0924 07:13:48.886335 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.886385 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:13:48.886577 13866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44705: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44705: connect: Connection refused (error 111)
I20260924 07:13:48.887100 13872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:48.888211 13872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:48.896586 13881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.896639 13872 server_base.cc:1034] running on GCE node
W20260924 07:13:48.896586 13878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:48.896588 13879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.897482 13872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:48.898634 13872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:48.899801 13872 hybrid_clock.cc:648] HybridClock initialized: now 1790234028899781 us; error 36 us; skew 500 ppm
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:48.902484 13872 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:13:48.903436 13872 webserver.cc:460] Webserver started at http://127.0.98.60:46497/ using document root <none> and password file <none>
I20260924 07:13:48.903976 13872 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:48.904044 13872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:48.904247 13872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:48.906294 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data/instance:
uuid: "1dddd40270624a46a090ee84803d8980"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "01a479ee9a8a6e25bd95bab768bc2ebb"
server_key_iv: "0e8e0cec4986c1d6f7ee6025e8b62658"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.906857 13872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal/instance:
uuid: "1dddd40270624a46a090ee84803d8980"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "01a479ee9a8a6e25bd95bab768bc2ebb"
server_key_iv: "0e8e0cec4986c1d6f7ee6025e8b62658"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.910740 13872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:13:48.913340 13888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:48.914775 13872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:13:48.914906 13872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "1dddd40270624a46a090ee84803d8980"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-cnv3"
server_key: "01a479ee9a8a6e25bd95bab768bc2ebb"
server_key_iv: "0e8e0cec4986c1d6f7ee6025e8b62658"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.915011 13872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:48.931295 13872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:48.934600 13872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:48.934789 13872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:48.941440 13872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44705
I20260924 07:13:48.941522 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44705 every 8 connection(s)
I20260924 07:13:48.942433 13872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:13:48.945477 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:48.949455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 13872
I20260924 07:13:48.949548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233866113376-392-0/minicluster-data/master-2/wal/instance
I20260924 07:13:48.949410 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.949806 392 external_mini_cluster.cc:1428] Setting key 2b8e53c4b0a0440f97bf909d42960491
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:48.958858 13791 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 3
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:48.964006 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.951898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.98.62:46105)
0924 07:13:48.952029 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.952033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.952176 (+ 143us) server_negotiation.cc:407] Connection header received
0924 07:13:48.952582 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.952585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.952635 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.952718 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.953751 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.954541 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.956082 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.956503 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.956613 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.959515 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.959519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.959522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.959550 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.961413 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.962103 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.962107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.962108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.962197 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.962554 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.962557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.962559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.962748 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.962843 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.963470 (+ 627us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.963589 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:13:48.964823 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.951052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.0.98.60:44705)
0924 07:13:48.951485 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.951500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.951532 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:13:48.952316 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.952338 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.952552 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.952725 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.953616 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.954817 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.956082 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.956666 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.956852 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.960069 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.960076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.960092 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.960115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.961955 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.962364 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.962370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.962372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.962469 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.962756 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.962760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.962761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.963182 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.963445 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.963644 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.963933 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
I20260924 07:13:48.965073 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.951722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46105 (local address 127.0.0.1:44844)
0924 07:13:48.952038 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.952086 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.952382 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.952779 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.952785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.953226 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.953595 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.953614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.954703 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.954706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.955958 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.955968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.956676 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.956680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.956864 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.957586 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.957607 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.959341 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.961521 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.961529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.961540 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.961985 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.962297 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.962300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.962302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.962466 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.962888 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.962895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.963206 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.964204 (+ 998us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.964384 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":48,"thread_start_us":78,"threads_started":1}
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:48.966825 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:48.968851 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.959325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35760)
0924 07:13:48.959957 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.959973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.960055 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.960679 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.960682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.960960 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.961075 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.961083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.961955 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.961958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.962733 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.962744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.963347 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.963351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.963462 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.963985 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.963997 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.965439 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.967306 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.967309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.967311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.967575 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.967857 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.967860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.967862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.967948 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.968339 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.968342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.968428 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.968551 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.968687 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260924 07:13:48.969089 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.959637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.0.98.60:44705)
0924 07:13:48.960028 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.960036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.960048 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:13:48.960187 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.960190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.960239 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.960317 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.961191 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.961841 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.962836 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.963202 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.963663 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.965569 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.965572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.965574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.965593 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.967204 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.967666 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.967669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.967671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.967758 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.968054 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.968057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.968059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.968250 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.968332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.968614 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.968732 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":186,"threads_started":1}
I20260924 07:13:48.971923 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: Bootstrap starting.
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:48.974639 13791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:48.975510 13791 log.cc:824] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:48.977145 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.967437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.98.61:34469)
0924 07:13:48.967554 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.967557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.967770 (+ 213us) server_negotiation.cc:407] Connection header received
0924 07:13:48.967957 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.967971 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.968035 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.968089 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.968840 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.969579 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.970617 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.970946 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.971018 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.973308 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.973312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.973315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.973344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.974847 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.975498 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.975501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.975503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.975592 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.976044 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.976047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.976049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.976345 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.976449 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.976849 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.976972 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:13:48.977260 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.968040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.0.98.62:46105)
0924 07:13:48.968222 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.968225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.968234 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:13:48.968259 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.968262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.968291 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.968372 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.969638 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.970391 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.971435 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.971746 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.971845 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.974311 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.974314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.974316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.974339 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.975684 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.976058 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.976062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.976064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.976147 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.976429 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.976432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.976434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.976641 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.976826 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.977018 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.977122 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 07:13:48.977589 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.967277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34469 (local address 127.0.0.1:32832)
0924 07:13:48.967371 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.967640 (+ 269us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.967844 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.968494 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.968497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.968618 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.968713 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.968717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.969693 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.969697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.970498 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.970508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.971161 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.971164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.971270 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.971714 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.971728 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.973147 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.975129 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.975133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.975135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.975405 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.975703 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.975706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.975708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.975910 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.976511 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.976513 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.976606 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.977340 (+ 734us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.977470 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1}
I20260924 07:13:48.977713 13866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } attempt: 3
I20260924 07:13:48.978626 13791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: No bootstrap required, opened a new log
I20260924 07:13:48.978996 13941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:48.984087 13791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:48.984562 13791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:48.984619 13791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd418d0c4004026bedb7869f176e0eb, State: Initialized, Role: FOLLOWER
I20260924 07:13:48.985164 13791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
Sep 24 07:13:48 dist-test-slave-cnv3 krb5kdc[13710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:13:48.986882 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.978213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35764)
0924 07:13:48.978458 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.978472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.978539 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.978776 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.978779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.978941 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.979042 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.979049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.980011 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.980014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.980710 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.980720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.981262 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.981266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.981351 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.981806 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.981819 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.983351 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.985235 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.985238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.985240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.985473 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.985792 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.985795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.985797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.985899 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.986401 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.986404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.986501 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.986577 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.986710 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260924 07:13:48.987493 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.98.60:44705)
0924 07:13:48.978549 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.978553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.978586 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:13:48.978619 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.978622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.978679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.978746 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.979171 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.979918 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.980801 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.981138 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.981557 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.983483 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.983487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.983490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.983514 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.985119 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.985590 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.985593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.985595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.985679 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.986009 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.986012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.986014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.986291 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.987150 (+ 859us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.987222 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.987330 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:13:48.987666 13791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:48.988353 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.979448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44705 (local address 127.0.0.1:35776)
0924 07:13:48.979746 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:48.979758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:48.979820 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:48.980411 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:48.980414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:48.980571 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:48.980656 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.980662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.981608 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.981611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.982421 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:48.982430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.983033 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.983044 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:48.983114 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.983565 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:48.983578 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:48.984990 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:48.986728 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.986731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.986733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.987054 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.987310 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:48.987312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:48.987314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:48.987411 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:48.987938 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:48.987941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:48.988014 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 07:13:48.988093 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:13:48.988208 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260924 07:13:48.988811 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.979596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.0.98.60:44705)
0924 07:13:48.980054 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.980058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.980067 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:13:48.980092 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.980094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.980127 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.980384 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.980761 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.981502 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.982536 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.982870 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.982950 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.985109 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.985113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.985115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.985137 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.986617 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.987151 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.987154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.987155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.987227 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.987494 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.987497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.987499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.987827 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.987907 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.988563 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.988678 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260924 07:13:48.989689 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.978721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.61:34469)
0924 07:13:48.978820 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.978823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.979038 (+ 215us) server_negotiation.cc:407] Connection header received
0924 07:13:48.979193 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.979197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.979235 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.979279 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.980069 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.981502 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.982548 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.982870 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:48.982937 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:48.985920 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.985924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.985927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:48.985952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.987534 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.988094 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.988097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.988099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.988188 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:48.988596 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:48.988600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:48.988601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:48.988795 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.988978 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.989441 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.989550 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":679,"server-negotiator.queue_time_us":42}
I20260924 07:13:48.990756 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: Bootstrap starting.
W20260924 07:13:48.993440 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:48.993512 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:48.993569 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: Bootstrap starting.
I20260924 07:13:48.993870 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:48.992781 13723 master_runner.cc:186] Error getting master registration for 127.0.98.61:34469: OK, instance_id { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" instance_seqno: 1790234028722539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:13:48.997001 13723 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:13:48.997443 13866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:48.998337 13866 log.cc:824] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:48.999640 13941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:49.000487 13941 log.cc:824] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:49.001088 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.001226 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.005734 13866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: No bootstrap required, opened a new log
I20260924 07:13:49.005844 13941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: No bootstrap required, opened a new log
I20260924 07:13:49.007407 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.996095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.0.98.61:34469)
0924 07:13:48.996216 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.996220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.997574 (+ 1354us) server_negotiation.cc:407] Connection header received
0924 07:13:48.997802 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.997805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.997849 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.997897 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.998907 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.999674 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.000994 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.001295 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.001378 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:49.003004 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:49.003008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:49.003011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:49.003036 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:49.004823 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.006063 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.006066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.006069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.006154 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.006448 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.006451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.006453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.006658 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:49.006767 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:49.007094 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:13:49.007212 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:13:49.009891 13866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.010123 13866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:49.010180 13866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b3a1124211f44d9b867bf1a279d3119, State: Initialized, Role: FOLLOWER
I20260924 07:13:49.010664 13866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.010869 13941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.011102 13941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:49.011154 13941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dddd40270624a46a090ee84803d8980, State: Initialized, Role: FOLLOWER
I20260924 07:13:49.011663 13941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.012151 13866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:49.012892 13941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:49.017443 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.996800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.98.62:46105)
0924 07:13:48.996909 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.996913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.997580 (+ 667us) server_negotiation.cc:407] Connection header received
0924 07:13:48.997780 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.997783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.997831 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.997881 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.998840 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:48.999679 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.000957 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.001302 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.001383 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:49.003157 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:49.003161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:49.003164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:49.003189 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:49.005504 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.015819 (+ 10315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.015823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.015825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.015917 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.016342 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.016345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.016347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.016566 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:49.016688 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:49.017147 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 07:13:49.017285 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:13:49.017693 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.997223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.0.98.60:44705)
0924 07:13:48.997333 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.997336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:49.008293 (+ 10957us) server_negotiation.cc:407] Connection header received
0924 07:13:49.008463 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:49.008467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:49.008516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:49.008562 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:49.009293 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.009996 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.011327 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.011654 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.011730 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:49.013509 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:49.013514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:49.013517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:49.013540 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:49.015791 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.016235 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.016238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.016240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.016337 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.016624 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.016627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.016629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.016828 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:49.016917 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:49.017439 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 07:13:49.017555 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:13:49.019562 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.019738 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.020040 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.020164 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.022230 13798 master_runner.cc:263] No leader master found from master 6b3a1124211f44d9b867bf1a279d3119
I20260924 07:13:49.022297 13798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:13:49.022308 13872 master_runner.cc:263] No leader master found from master 1dddd40270624a46a090ee84803d8980
I20260924 07:13:49.022373 13872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:13:49.023610 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:49.023696 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:49.023902 13971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:49.024036 13971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.024891 13971 leader_election.cc:288] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dd418d0c4004026bedb7869f176e0eb (127.0.98.62:46105), 6b3a1124211f44d9b867bf1a279d3119 (127.0.98.61:34469)
I20260924 07:13:49.025295 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1dddd40270624a46a090ee84803d8980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd418d0c4004026bedb7869f176e0eb" is_pre_election: true
I20260924 07:13:49.025585 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dddd40270624a46a090ee84803d8980 in term 0.
I20260924 07:13:49.025993 13890 leader_election.cc:302] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dddd40270624a46a090ee84803d8980, 2dd418d0c4004026bedb7869f176e0eb; no voters:
I20260924 07:13:49.026208 13971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:49.026293 13971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:49.026341 13971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:49.026489 13839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1dddd40270624a46a090ee84803d8980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b3a1124211f44d9b867bf1a279d3119" is_pre_election: true
I20260924 07:13:49.026755 13839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dddd40270624a46a090ee84803d8980 in term 0.
I20260924 07:13:49.027608 13971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.028136 13971 leader_election.cc:288] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [CANDIDATE]: Term 1 election: Requested vote from peers 2dd418d0c4004026bedb7869f176e0eb (127.0.98.62:46105), 6b3a1124211f44d9b867bf1a279d3119 (127.0.98.61:34469)
I20260924 07:13:49.028198 13839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1dddd40270624a46a090ee84803d8980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b3a1124211f44d9b867bf1a279d3119"
I20260924 07:13:49.028184 13765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1dddd40270624a46a090ee84803d8980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd418d0c4004026bedb7869f176e0eb"
I20260924 07:13:49.028303 13839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:49.028317 13765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:49.029259 13765 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dddd40270624a46a090ee84803d8980 in term 1.
I20260924 07:13:49.029578 13839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dddd40270624a46a090ee84803d8980 in term 1.
I20260924 07:13:49.029717 13890 leader_election.cc:302] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dddd40270624a46a090ee84803d8980, 2dd418d0c4004026bedb7869f176e0eb; no voters:
I20260924 07:13:49.029875 13971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:49.030138 13971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [term 1 LEADER]: Becoming Leader. State: Replica: 1dddd40270624a46a090ee84803d8980, State: Running, Role: LEADER
I20260924 07:13:49.030462 13971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } }
I20260924 07:13:49.032367 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dddd40270624a46a090ee84803d8980. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.032482 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: This master's current role is: LEADER
W20260924 07:13:49.035182 13993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:49.035315 13993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:49.043141 13997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:49.044231 13997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:49.052235 13839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [term 1 FOLLOWER]: Refusing update from remote peer 1dddd40270624a46a090ee84803d8980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:49.052264 13765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [term 1 FOLLOWER]: Refusing update from remote peer 1dddd40270624a46a090ee84803d8980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:49.052932 13971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:49.053371 13971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:49.057545 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dddd40270624a46a090ee84803d8980. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.057693 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.062330 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dddd40270624a46a090ee84803d8980. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.062454 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.075438 13999 mvcc.cc:205] Tried to move back new op lower bound from 7332798582988525568 to 7332798582912094208. Current Snapshot: MvccSnapshot[applied={T|T < 7332798582988525568}]
I20260924 07:13:49.076643 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.076668 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.076790 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.076821 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:49.076968 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.077093 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:49.077339 13971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dddd40270624a46a090ee84803d8980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd418d0c4004026bedb7869f176e0eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46105 } } peers { permanent_uuid: "6b3a1124211f44d9b867bf1a279d3119" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34469 } } peers { permanent_uuid: "1dddd40270624a46a090ee84803d8980" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44705 } } }
I20260924 07:13:49.077442 13971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:49.077869 14003 mvcc.cc:205] Tried to move back new op lower bound from 7332798582988525568 to 7332798582912094208. Current Snapshot: MvccSnapshot[applied={T|T < 7332798582988525568}]
I20260924 07:13:49.087476 14001 mvcc.cc:205] Tried to move back new op lower bound from 7332798582988525568 to 7332798582912094208. Current Snapshot: MvccSnapshot[applied={T|T < 7332798582988525568 or (T in {7332798582988525568})}]
I20260924 07:13:49.088088 13997 catalog_manager.cc:1347] Generated new cluster ID: acf105909abf4cb29ab36c2c247e34ff
I20260924 07:13:49.088166 13997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:49.160936 13997 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:49.162261 13997 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:49.181192 13997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dddd40270624a46a090ee84803d8980: Generated new TSK 0
I20260924 07:13:49.182188 13997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:49.994856 13959 catalog_manager.cc:1259] Loaded cluster ID: acf105909abf4cb29ab36c2c247e34ff
I20260924 07:13:49.994936 13959 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: loading cluster ID for follower catalog manager: success
I20260924 07:13:49.996618 13959 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: acquiring CA information for follower catalog manager: success
I20260924 07:13:49.997788 13959 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2dd418d0c4004026bedb7869f176e0eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:13:50.025095 13987 catalog_manager.cc:1259] Loaded cluster ID: acf105909abf4cb29ab36c2c247e34ff
I20260924 07:13:50.025185 13987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: loading cluster ID for follower catalog manager: success
I20260924 07:13:50.026829 13987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: acquiring CA information for follower catalog manager: success
I20260924 07:13:50.028082 13987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b3a1124211f44d9b867bf1a279d3119: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:13:50.252043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13723
I20260924 07:13:50.257882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13798
I20260924 07:13:50.263064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 13872
2026-09-24T07:13:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3054 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:50 dist-test-slave-cnv3 krb5kdc[14014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:50 dist-test-slave-cnv3 krb5kdc[14014](info): set up 2 sockets
Sep 24 07:13:50 dist-test-slave-cnv3 krb5kdc[14014](info): commencing operation
krb5kdc: starting...
W20260924 07:13:51.219530 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:51.342504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:51.454676 14027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:51.455020 14027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:51.455081 14027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:51.459293 14027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:51.459406 14027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:51.459436 14027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:51.459472 14027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:51.459502 14027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:51.465179 14027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:51.466351 14027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:51.467460 14027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:51.476284 14035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:51.476329 14033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:51.476329 14032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:51.476648 14027 server_base.cc:1034] running on GCE node
I20260924 07:13:51.477232 14027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:51.478412 14027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:51.479589 14027 hybrid_clock.cc:648] HybridClock initialized: now 1790234031479544 us; error 56 us; skew 500 ppm
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:51.482652 14027 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:51.483721 14027 webserver.cc:460] Webserver started at http://127.0.98.62:35277/ using document root <none> and password file <none>
I20260924 07:13:51.484287 14027 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:51.484355 14027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:51.484541 14027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:51.486608 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "33c4e68306a743d5971ae6abe7348f3f"
format_stamp: "Formatted at 2026-09-24 07:13:51 on dist-test-slave-cnv3"
server_key: "2d0ed2d10550666ecdcebecbdc3046fb"
server_key_iv: "2b4a49b50ec2c08028d30eb8d42f22e9"
server_key_version: "encryptionkey@0"
I20260924 07:13:51.487154 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "33c4e68306a743d5971ae6abe7348f3f"
format_stamp: "Formatted at 2026-09-24 07:13:51 on dist-test-slave-cnv3"
server_key: "2d0ed2d10550666ecdcebecbdc3046fb"
server_key_iv: "2b4a49b50ec2c08028d30eb8d42f22e9"
server_key_version: "encryptionkey@0"
I20260924 07:13:51.490850 14027 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:51.493501 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:51.494833 14027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:51.494968 14027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "33c4e68306a743d5971ae6abe7348f3f"
format_stamp: "Formatted at 2026-09-24 07:13:51 on dist-test-slave-cnv3"
server_key: "2d0ed2d10550666ecdcebecbdc3046fb"
server_key_iv: "2b4a49b50ec2c08028d30eb8d42f22e9"
server_key_version: "encryptionkey@0"
I20260924 07:13:51.495150 14027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:51.510883 14027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:51.514204 14027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:51.514405 14027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:51.521382 14027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35517
I20260924 07:13:51.521449 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35517 every 8 connection(s)
I20260924 07:13:51.522342 14027 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:51.525198 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:51.528745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14027
I20260924 07:13:51.528844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:51.529165 392 external_mini_cluster.cc:1428] Setting key 0724f8fb2f7a4c44e7e494e1f61a6cd1
I20260924 07:13:51.531219 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: Bootstrap starting.
I20260924 07:13:51.533484 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:51.534209 14095 log.cc:824] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:51.536092 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: No bootstrap required, opened a new log
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:51.540457 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260924 07:13:51.540743 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:51.540815 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c4e68306a743d5971ae6abe7348f3f, State: Initialized, Role: FOLLOWER
I20260924 07:13:51.541326 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260924 07:13:51.541487 14095 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:51.541555 14095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:51.541653 14095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:51.542778 14095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260924 07:13:51.543155 14095 leader_election.cc:302] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c4e68306a743d5971ae6abe7348f3f; no voters:
I20260924 07:13:51.543526 14095 leader_election.cc:288] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:51.543641 14100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:51.543795 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:51.530499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.0.98.62:35517)
0924 07:13:51.530957 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:51.531038 (+ 81us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:51.531076 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:13:51.531767 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:51.531786 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:51.532166 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:51.532490 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:51.533264 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:51.534284 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:51.535142 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:51.535659 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:51.535799 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:51.538433 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:51.538439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:51.538465 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:51.538501 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:51.540487 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:51.540968 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:51.540975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:51.540980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:51.541122 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:51.541386 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:51.541389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:51.541391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:51.541948 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:51.542135 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:51.542432 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:13:51.542752 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260924 07:13:51.543840 14100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [term 1 LEADER]: Becoming Leader. State: Replica: 33c4e68306a743d5971ae6abe7348f3f, State: Running, Role: LEADER
I20260924 07:13:51.544268 14100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } }
I20260924 07:13:51.544813 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:51.546407 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c4e68306a743d5971ae6abe7348f3f. Latest consensus state: current_term: 1 leader_uuid: "33c4e68306a743d5971ae6abe7348f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } } }
I20260924 07:13:51.546550 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:51.546886 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 07:13:51.549995 14115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:51.550066 14115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:51.550233 14111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:51.550220 14102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c4e68306a743d5971ae6abe7348f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c4e68306a743d5971ae6abe7348f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35517 } } }
I20260924 07:13:51.550313 14102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:51.551247 14111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:51.557039 14111 catalog_manager.cc:1347] Generated new cluster ID: 75c12f926dc4447090a23d95d490df09
I20260924 07:13:51.557121 14111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:51.569499 14111 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:51.570827 14111 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:51.581255 14111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c4e68306a743d5971ae6abe7348f3f: Generated new TSK 0
I20260924 07:13:51.582192 14111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:51.602777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14027
2026-09-24T07:13:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1337 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14122](info): set up 2 sockets
Sep 24 07:13:51 dist-test-slave-cnv3 krb5kdc[14122](info): commencing operation
krb5kdc: starting...
W20260924 07:13:52.547461 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:52 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:52.670463 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:52.783950 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:52.784248 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:52.784308 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:52.788422 14135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:52.788494 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:52.788522 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:52.788542 14135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:52.788571 14135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:52.794130 14135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:52.795259 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:52.796331 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:13:52.804929 14135 server_base.cc:1034] running on GCE node
W20260924 07:13:52.804869 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:52.804862 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:52.804858 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:52.805776 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:52.806942 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:52.808123 14135 hybrid_clock.cc:648] HybridClock initialized: now 1790234032808101 us; error 36 us; skew 500 ppm
Sep 24 07:13:52 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:52.814275 14135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:52.815320 14135 webserver.cc:460] Webserver started at http://127.0.98.62:38121/ using document root <none> and password file <none>
I20260924 07:13:52.815917 14135 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:52.815984 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:52.816202 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:52.818337 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "d9e79f91e38849e18419ab355fbb14b4"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-cnv3"
server_key: "a9e702b2ac186f26fa923fe1cdc4f3b6"
server_key_iv: "ea27f3b2b6d28b574a6fa92eaaf3765d"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.818897 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9e79f91e38849e18419ab355fbb14b4"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-cnv3"
server_key: "a9e702b2ac186f26fa923fe1cdc4f3b6"
server_key_iv: "ea27f3b2b6d28b574a6fa92eaaf3765d"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.822721 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:13:52.825275 14150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:52.826723 14135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:13:52.826844 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "d9e79f91e38849e18419ab355fbb14b4"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-cnv3"
server_key: "a9e702b2ac186f26fa923fe1cdc4f3b6"
server_key_iv: "ea27f3b2b6d28b574a6fa92eaaf3765d"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.826967 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:52.838658 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:52.841907 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:52.842082 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:52.849010 14135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41549
I20260924 07:13:52.849272 14202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41549 every 8 connection(s)
I20260924 07:13:52.850018 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:52.852998 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:52.856985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14135
I20260924 07:13:52.857093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:52.857353 392 external_mini_cluster.cc:1428] Setting key 83cd28988632450cd0b815cbe7eed99c
I20260924 07:13:52.859289 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap starting.
I20260924 07:13:52.861985 14203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:52.862757 14203 log.cc:824] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:52.865106 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: No bootstrap required, opened a new log
Sep 24 07:13:52 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:52.869309 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:52.869558 14203 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:52.869593 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Initialized, Role: FOLLOWER
I20260924 07:13:52.870038 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:52.870157 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:52.870213 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:52.870308 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:52.871294 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:52.871706 14203 leader_election.cc:302] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e79f91e38849e18419ab355fbb14b4; no voters:
I20260924 07:13:52.872044 14203 leader_election.cc:288] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:52.872138 14208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:52.872323 14208 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 LEADER]: Becoming Leader. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Running, Role: LEADER
I20260924 07:13:52.872684 14208 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:52.872767 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:52.859055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42250 (local address 127.0.98.62:41549)
0924 07:13:52.859546 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:52.859563 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:52.859602 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:13:52.860297 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:52.860316 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:52.860682 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:52.861002 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:52.861956 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:52.863135 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:52.864438 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:52.864931 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:52.865059 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:52.867861 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:52.867867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:52.867882 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:52.867911 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:52.869593 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:52.870076 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:52.870082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:52.870086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:52.870242 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:52.870497 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:52.870501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:52.870503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:52.870932 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:52.871132 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:52.871456 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 07:13:52.871761 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":124,"threads_started":1}
I20260924 07:13:52.873405 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:52.873945 14208 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e79f91e38849e18419ab355fbb14b4. Latest consensus state: current_term: 1 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:52.874060 14208 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:52.874873 14213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:52.875310 14209 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:52.875442 14209 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:52.877991 14213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:52.883497 14213 catalog_manager.cc:1347] Generated new cluster ID: 2a9ae31dfeb346f8be70f7bf15b7ee3b
I20260924 07:13:52.883580 14213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:52.897424 14213 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:52.898424 14213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:52.908027 14213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Generated new TSK 0
I20260924 07:13:52.908753 14213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:13:52.942531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:53.056919 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:53.057214 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:53.057260 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:53.061455 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:53.061533 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:53.061595 14229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:53.068082 14229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:53.069718 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:53.071151 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:53.082118 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:53.082118 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:53.082105 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.082679 14229 server_base.cc:1034] running on GCE node
I20260924 07:13:53.083134 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:53.083841 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:53.085016 14229 hybrid_clock.cc:648] HybridClock initialized: now 1790234033084994 us; error 35 us; skew 500 ppm
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:53.087844 14229 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:53.088789 14229 webserver.cc:460] Webserver started at http://127.0.98.1:43059/ using document root <none> and password file <none>
I20260924 07:13:53.089406 14229 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:53.089478 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:53.089671 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:53.091815 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "b73858d5e3ce4fafb4257faa98771f7c"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "f17c507c49534d5f6e0b019b90aa5350"
server_key_iv: "d3cb48a084213991019e475ac2c41a9c"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.092381 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b73858d5e3ce4fafb4257faa98771f7c"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "f17c507c49534d5f6e0b019b90aa5350"
server_key_iv: "d3cb48a084213991019e475ac2c41a9c"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.096282 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:13:53.098922 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.100437 14229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:53.100566 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "b73858d5e3ce4fafb4257faa98771f7c"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "f17c507c49534d5f6e0b019b90aa5350"
server_key_iv: "d3cb48a084213991019e475ac2c41a9c"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.100683 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:53.119781 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:53.123168 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:53.123404 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:53.124047 14229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:53.124940 14229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:53.124990 14229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.125046 14229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:53.125075 14229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.134130 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37647
I20260924 07:13:53.134195 14357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37647 every 8 connection(s)
I20260924 07:13:53.135107 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:53.139142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14229
I20260924 07:13:53.139266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:53.139624 392 external_mini_cluster.cc:1428] Setting key db567a566379677544212bb1ba80797a
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:13:53.150102 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.137045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46933 (local address 127.0.98.62:41549)
0924 07:13:53.137179 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.137183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.138077 (+ 894us) server_negotiation.cc:407] Connection header received
0924 07:13:53.139594 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.139598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.139656 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.139730 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.141069 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.141834 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.143025 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.143260 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.143349 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.145879 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.145883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.145885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.145909 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.147498 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.148537 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.148541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.148543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.148624 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.149032 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.149034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.149036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.149223 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.149294 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.149851 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.149968 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:13:53.151028 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.137437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:46933)
0924 07:13:53.137926 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.137971 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.139260 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.139916 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.139923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.140408 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:53.140925 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.140944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.141962 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.141965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.142913 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.142922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.143435 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.143440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.143614 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.144193 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:53.144209 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:53.145699 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:53.148134 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.148139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.148156 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.148436 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.148728 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.148732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.148734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.148939 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.149300 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:53.149305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:53.149580 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.149785 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.150080 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:13:53.152550 14358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:53.152861 14358 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:53.153614 14358 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:53.155474 14167 ts_manager.cc:194] Registered new tserver with Master: b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647)
I20260924 07:13:53.157140 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:13:53.170259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:53.286409 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:53.286705 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:53.286753 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:53.290921 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:53.290997 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:53.291057 14363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:53.296717 14363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:53.297909 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:53.298962 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:53.309211 14368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:53.309283 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.309267 14363 server_base.cc:1034] running on GCE node
W20260924 07:13:53.309211 14369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.309989 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:53.310711 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:53.311879 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790234033311860 us; error 34 us; skew 500 ppm
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:53.314828 14363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:53.315864 14363 webserver.cc:460] Webserver started at http://127.0.98.2:44975/ using document root <none> and password file <none>
I20260924 07:13:53.316445 14363 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:53.316514 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:53.316704 14363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:53.318794 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "4828c87e53604025ef055e5daf32c625"
server_key_iv: "54e25747a4566300d70d6f393ac575b7"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.319341 14363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "4828c87e53604025ef055e5daf32c625"
server_key_iv: "54e25747a4566300d70d6f393ac575b7"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.323241 14363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:13:53.325986 14378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.327358 14363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:53.327543 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "4828c87e53604025ef055e5daf32c625"
server_key_iv: "54e25747a4566300d70d6f393ac575b7"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.327646 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:53.352073 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:53.355557 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:53.355751 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:53.356374 14363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:53.357265 14363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:53.357311 14363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.357365 14363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:53.357395 14363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.366142 14363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41473
I20260924 07:13:53.366165 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41473 every 8 connection(s)
I20260924 07:13:53.367067 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:53.377084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14363
I20260924 07:13:53.377194 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:53.377431 392 external_mini_cluster.cc:1428] Setting key 6202e254794a6a0fc52f74778518ec0f
I20260924 07:13:53.381281 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.368947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58117 (local address 127.0.98.62:41549)
0924 07:13:53.369084 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.369088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.369999 (+ 911us) server_negotiation.cc:407] Connection header received
0924 07:13:53.371339 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.371341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.371426 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.371507 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.372888 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.373484 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.374747 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.374977 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.375055 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.377634 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.377637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.377639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.377664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.379119 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.379696 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.379700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.379702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.379780 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.380104 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.380107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.380109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.380318 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.380404 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.380946 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.381097 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:13:53.382375 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.369359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:58117)
0924 07:13:53.369844 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.369887 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.371128 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.371720 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.371729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.372195 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:53.372760 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.372775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.373585 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.373588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.374642 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.374652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.375108 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.375111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.375321 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.375958 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:53.375974 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:53.377465 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:53.379255 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.379262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.379278 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.379600 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.379862 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.379865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.379867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.380018 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.380418 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:53.380426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:53.380706 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.380892 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.381259 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260924 07:13:53.383599 14492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:53.383850 14492 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:53.384361 14492 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:53.385432 14167 ts_manager.cc:194] Registered new tserver with Master: abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
I20260924 07:13:53.386032 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58117
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:13:53.406126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:53.518517 14497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:53.518802 14497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:53.518863 14497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:53.523063 14497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:53.523139 14497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:53.523224 14497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:53.528760 14497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:53.529896 14497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:53.531024 14497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:53.541016 14502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.541157 14497 server_base.cc:1034] running on GCE node
W20260924 07:13:53.541018 14505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:53.541096 14503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.541676 14497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:53.542342 14497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:53.543503 14497 hybrid_clock.cc:648] HybridClock initialized: now 1790234033543474 us; error 46 us; skew 500 ppm
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:53.546177 14497 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:53.547057 14497 webserver.cc:460] Webserver started at http://127.0.98.3:40105/ using document root <none> and password file <none>
I20260924 07:13:53.547657 14497 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:53.547719 14497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:53.547896 14497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:53.549932 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd2eaf3272a944aeae87aa918b61de75"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "e394180ec6000af28033307113ecd3bc"
server_key_iv: "29270bd7768be66f85b496ff35275f74"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.550446 14497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd2eaf3272a944aeae87aa918b61de75"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "e394180ec6000af28033307113ecd3bc"
server_key_iv: "29270bd7768be66f85b496ff35275f74"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.554194 14497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:13:53.556766 14512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.558110 14497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:53.558283 14497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "bd2eaf3272a944aeae87aa918b61de75"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "e394180ec6000af28033307113ecd3bc"
server_key_iv: "29270bd7768be66f85b496ff35275f74"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.558393 14497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:53.570523 14497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:53.573832 14497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:53.574005 14497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:53.574584 14497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:53.575476 14497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:53.575531 14497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.575577 14497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:53.575596 14497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.584251 14497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42747
I20260924 07:13:53.584275 14625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42747 every 8 connection(s)
I20260924 07:13:53.585163 14497 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:53.593380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14497
I20260924 07:13:53.593508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:53.593770 392 external_mini_cluster.cc:1428] Setting key c9be3224ec2a20d8aa191a5b39c6f996
Sep 24 07:13:53 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:53.598537 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.586994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33473 (local address 127.0.98.62:41549)
0924 07:13:53.587149 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.587152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.588089 (+ 937us) server_negotiation.cc:407] Connection header received
0924 07:13:53.589467 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.589471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.589530 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.589604 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.590812 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.591334 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.592514 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.592744 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.592835 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.595532 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.595535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.595538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.595562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.596817 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.597263 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.597265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.597266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.597331 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.597592 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.597594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.597595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.597763 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.597834 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.598276 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.598383 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:13:53.599498 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.587353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:33473)
0924 07:13:53.587919 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.587970 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.589255 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.589802 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.589809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.590166 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:53.590669 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.590682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.591487 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.591490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.592394 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.592403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.592889 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.592893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.593065 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.593703 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:53.593720 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:53.595343 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:53.596916 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.596921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.596935 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.597190 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.597395 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:53.597397 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:53.597398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:53.597527 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:53.597834 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:53.597838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:53.598073 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.598222 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.598562 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":113,"threads_started":1}
I20260924 07:13:53.600648 14626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:53.600854 14626 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:53.601318 14626 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:53.602352 14167 ts_manager.cc:194] Registered new tserver with Master: bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747)
I20260924 07:13:53.602934 14167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33473
I20260924 07:13:53.607590 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:53.616539 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.609512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.0.98.62:41549)
0924 07:13:53.609632 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.609636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.609827 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:13:53.609927 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.609930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.609966 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.610028 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.610645 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.611161 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.612093 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.612303 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.612377 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.613559 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.613562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.613564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.613585 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.614862 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.615294 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.615297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.615299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.615458 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.615714 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.615716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.615718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.615954 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.616056 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.616301 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.616394 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:13:53.619447 14167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:53.622731 14167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:53.638183 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.631772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.0.1:52702)
0924 07:13:53.632188 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.632246 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.632379 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.633467 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.633473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.633494 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:53.633648 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.633654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.634974 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.634977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.636518 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.636529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.637517 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.637521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.637699 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.637744 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.637862 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.638034 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":155,"thread_start_us":107,"threads_started":1}
I20260924 07:13:53.638188 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.632095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.0.98.2:41473)
0924 07:13:53.632977 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.632989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.633013 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:13:53.633095 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.633100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.633323 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.633472 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:53.633767 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.634860 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.636621 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.637395 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.637500 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.637648 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.637915 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.638016 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":113,"threads_started":1}
I20260924 07:13:53.638258 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.632072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.0.98.1:37647)
0924 07:13:53.632673 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.632688 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.632712 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:13:53.633389 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.633396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.633596 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.633742 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:53.634192 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.635213 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.636542 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.637389 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.637500 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.637648 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.637984 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.638124 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":96,"threads_started":1}
I20260924 07:13:53.638978 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.631772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:57042)
0924 07:13:53.632637 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.632649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.633208 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.633959 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.633962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.633980 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:53.634067 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.634073 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.635328 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.635332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.636431 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.636440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.637517 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.637521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.637717 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.637744 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.637879 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.638840 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":1516,"thread_start_us":115,"threads_started":1}
I20260924 07:13:53.639655 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.632747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42747 (local address 127.0.0.1:46858)
0924 07:13:53.633020 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.633033 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.633114 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.633728 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.633731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.633749 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:53.633829 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.633834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.634926 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.634930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.636518 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.636529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.639245 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.639248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.639342 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.639364 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.639469 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.639538 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":84,"thread_start_us":90,"threads_started":1}
I20260924 07:13:53.640156 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.632907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.98.3:42747)
0924 07:13:53.633358 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.633370 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.633390 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:13:53.633453 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.633457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.633595 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.633742 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:53.633960 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.634796 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.637716 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.638489 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.638583 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.638959 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.639919 (+ 960us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.640019 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":121,"threads_started":1}
I20260924 07:13:53.641548 14426 tablet_service.cc:1467] Processing CreateTablet for tablet f9e5c3d57f4042edbab88e60279f28bc (DEFAULT_TABLE table=test-table [id=54720e24c40f4e668013b33b0ebd5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:53.641716 14292 tablet_service.cc:1467] Processing CreateTablet for tablet f9e5c3d57f4042edbab88e60279f28bc (DEFAULT_TABLE table=test-table [id=54720e24c40f4e668013b33b0ebd5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:53.642607 14292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e5c3d57f4042edbab88e60279f28bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:53.642658 14426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e5c3d57f4042edbab88e60279f28bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:53.642863 14560 tablet_service.cc:1467] Processing CreateTablet for tablet f9e5c3d57f4042edbab88e60279f28bc (DEFAULT_TABLE table=test-table [id=54720e24c40f4e668013b33b0ebd5dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:53.643935 14560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e5c3d57f4042edbab88e60279f28bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:53.648994 14643 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap starting.
I20260924 07:13:53.651242 14643 tablet_bootstrap.cc:654] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:53.651269 14645 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Bootstrap starting.
I20260924 07:13:53.652036 14643 log.cc:824] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:53.652612 14644 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Bootstrap starting.
I20260924 07:13:53.653343 14645 tablet_bootstrap.cc:654] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:53.654153 14643 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: No bootstrap required, opened a new log
I20260924 07:13:53.654361 14643 ts_tablet_manager.cc:1397] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:13:53.654604 14645 log.cc:824] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:53.655241 14644 tablet_bootstrap.cc:654] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:53.655972 14644 log.cc:824] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:53.656852 14645 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: No bootstrap required, opened a new log
I20260924 07:13:53.657053 14645 ts_tablet_manager.cc:1397] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:13:53.657671 14644 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: No bootstrap required, opened a new log
I20260924 07:13:53.657856 14644 ts_tablet_manager.cc:1397] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 07:13:53.658669 14643 raft_consensus.cc:348] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.658972 14643 raft_consensus.cc:374] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:53.659029 14643 raft_consensus.cc:729] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73858d5e3ce4fafb4257faa98771f7c, State: Initialized, Role: FOLLOWER
I20260924 07:13:53.659597 14643 consensus_queue.cc:260] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.660401 14358 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
I20260924 07:13:53.660778 14643 ts_tablet_manager.cc:1428] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:13:53.660997 14645 raft_consensus.cc:348] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.661231 14645 raft_consensus.cc:374] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:53.661270 14645 raft_consensus.cc:729] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2eaf3272a944aeae87aa918b61de75, State: Initialized, Role: FOLLOWER
I20260924 07:13:53.661294 14644 raft_consensus.cc:348] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.661643 14644 raft_consensus.cc:374] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:53.661711 14644 raft_consensus.cc:729] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc78d1c6cbf49e4b7c60b54e7fc2167, State: Initialized, Role: FOLLOWER
I20260924 07:13:53.661681 14645 consensus_queue.cc:260] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.662132 14644 consensus_queue.cc:260] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.662786 14645 ts_tablet_manager.cc:1428] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:13:53.662868 14492 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
I20260924 07:13:53.663190 14644 ts_tablet_manager.cc:1428] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260924 07:13:53.663532 14626 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
W20260924 07:13:53.836129 14627 tablet.cc:2369] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:53.868283 14493 tablet.cc:2369] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:53.886544 14359 tablet.cc:2369] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:53.940642 14649 raft_consensus.cc:482] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:53.940858 14649 raft_consensus.cc:504] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.942108 14649 leader_election.cc:288] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473), bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747)
I20260924 07:13:53.946233 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.942481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37259 (local address 127.0.98.2:41473)
0924 07:13:53.942629 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.942634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.942884 (+ 250us) server_negotiation.cc:407] Connection header received
0924 07:13:53.943339 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.943342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.943409 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.943481 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:53.943737 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.944225 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945272 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.945685 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945778 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.945840 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.946039 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.946113 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:13:53.946256 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.942386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.98.1:37259)
0924 07:13:53.942780 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.942794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.942912 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.943501 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.943504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.943520 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:53.943630 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.943635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.944329 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.944332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.945148 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.945157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.945800 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.945926 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.945948 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.946012 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.946086 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":28,"thread_start_us":102,"threads_started":1}
I20260924 07:13:53.946244 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.942922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45373 (local address 127.0.98.3:42747)
0924 07:13:53.943052 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.943055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.943068 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:13:53.943115 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.943117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.943165 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.943226 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:53.943642 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.944162 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945248 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.945674 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945766 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.945842 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.946057 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.946131 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:53.946270 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.942429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42747 (local address 127.0.98.1:45373)
0924 07:13:53.942829 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:53.942843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:53.942948 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:53.943281 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:53.943284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:53.943301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:53.943501 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.943507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.944268 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.944271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.945115 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:53.945124 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.945800 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.945803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:53.945943 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.945961 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:13:53.946026 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:13:53.946097 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":39,"thread_start_us":109,"threads_started":1}
I20260924 07:13:53.946936 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e5c3d57f4042edbab88e60279f28bc" candidate_uuid: "b73858d5e3ce4fafb4257faa98771f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" is_pre_election: true
I20260924 07:13:53.947011 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e5c3d57f4042edbab88e60279f28bc" candidate_uuid: "b73858d5e3ce4fafb4257faa98771f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2eaf3272a944aeae87aa918b61de75" is_pre_election: true
I20260924 07:13:53.947286 14446 raft_consensus.cc:2454] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73858d5e3ce4fafb4257faa98771f7c in term 0.
I20260924 07:13:53.947286 14580 raft_consensus.cc:2454] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b73858d5e3ce4fafb4257faa98771f7c in term 0.
I20260924 07:13:53.947809 14248 leader_election.cc:302] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc78d1c6cbf49e4b7c60b54e7fc2167, b73858d5e3ce4fafb4257faa98771f7c; no voters:
I20260924 07:13:53.948091 14649 raft_consensus.cc:2781] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:53.948179 14649 raft_consensus.cc:482] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:53.948226 14649 raft_consensus.cc:3037] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:53.949246 14649 raft_consensus.cc:504] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.949599 14649 leader_election.cc:288] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 1 election: Requested vote from peers abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473), bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747)
I20260924 07:13:53.950024 14446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e5c3d57f4042edbab88e60279f28bc" candidate_uuid: "b73858d5e3ce4fafb4257faa98771f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
I20260924 07:13:53.950024 14580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e5c3d57f4042edbab88e60279f28bc" candidate_uuid: "b73858d5e3ce4fafb4257faa98771f7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2eaf3272a944aeae87aa918b61de75"
I20260924 07:13:53.950160 14446 raft_consensus.cc:3037] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:53.950160 14580 raft_consensus.cc:3037] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:53.951486 14446 raft_consensus.cc:2454] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73858d5e3ce4fafb4257faa98771f7c in term 1.
I20260924 07:13:53.951486 14580 raft_consensus.cc:2454] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b73858d5e3ce4fafb4257faa98771f7c in term 1.
I20260924 07:13:53.951788 14248 leader_election.cc:302] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abc78d1c6cbf49e4b7c60b54e7fc2167, b73858d5e3ce4fafb4257faa98771f7c; no voters:
I20260924 07:13:53.951942 14649 raft_consensus.cc:2781] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:53.952157 14649 raft_consensus.cc:686] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 LEADER]: Becoming Leader. State: Replica: b73858d5e3ce4fafb4257faa98771f7c, State: Running, Role: LEADER
I20260924 07:13:53.952507 14649 consensus_queue.cc:237] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:53.955917 14165 catalog_manager.cc:5495] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c reported cstate change: term changed from 0 to 1, leader changed from <none> to b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b73858d5e3ce4fafb4257faa98771f7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:54.005728 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.002196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.0.98.1:37647)
0924 07:13:54.002335 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.002339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.002354 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:13:54.002432 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.002435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.002491 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.002591 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:54.002804 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.003294 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.004216 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.004463 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.004554 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.004853 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:54.005304 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:54.005420 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.005528 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.005584 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:13:54.012053 14580 raft_consensus.cc:1261] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Refusing update from remote peer b73858d5e3ce4fafb4257faa98771f7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:54.012053 14446 raft_consensus.cc:1261] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Refusing update from remote peer b73858d5e3ce4fafb4257faa98771f7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:54.012754 14654 consensus_queue.cc:1035] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:54.012933 14649 consensus_queue.cc:1035] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:54.022075 14660 mvcc.cc:205] Tried to move back new op lower bound from 7332798603305488384 to 7332798603071946752. Current Snapshot: MvccSnapshot[applied={T|T < 7332798603305488384}]
I20260924 07:13:54.030030 14165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:54.030246 14165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:54.033103 14165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Sending DeleteTablet for 3 replicas of tablet f9e5c3d57f4042edbab88e60279f28bc
I20260924 07:13:54.034026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14229
I20260924 07:13:54.034024 14426 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e5c3d57f4042edbab88e60279f28bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52702
I20260924 07:13:54.034631 14560 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e5c3d57f4042edbab88e60279f28bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46858
I20260924 07:13:54.035508 14668 tablet_replica.cc:330] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: stopping tablet replica
I20260924 07:13:54.035879 14668 raft_consensus.cc:2229] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:54.036252 14668 raft_consensus.cc:2258] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:54.037194 14667 tablet_replica.cc:330] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: stopping tablet replica
I20260924 07:13:54.037529 14667 raft_consensus.cc:2229] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:54.037848 14667 raft_consensus.cc:2258] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:54.038211 14668 ts_tablet_manager.cc:1905] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:54.039511 14667 ts_tablet_manager.cc:1905] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:54.040632 14668 ts_tablet_manager.cc:1918] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:54.040722 14668 log.cc:1196] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/f9e5c3d57f4042edbab88e60279f28bc
I20260924 07:13:54.041000 14668 ts_tablet_manager.cc:1939] T f9e5c3d57f4042edbab88e60279f28bc P bd2eaf3272a944aeae87aa918b61de75: Deleting consensus metadata
W20260924 07:13:54.041241 14153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37647 (error 108)
I20260924 07:13:54.042101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14363
I20260924 07:13:54.042203 14153 catalog_manager.cc:4841] TS bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): tablet f9e5c3d57f4042edbab88e60279f28bc (table test-table [id=54720e24c40f4e668013b33b0ebd5dbb]) successfully deleted
I20260924 07:13:54.042233 14667 ts_tablet_manager.cc:1918] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:54.042312 14667 log.cc:1196] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/f9e5c3d57f4042edbab88e60279f28bc
I20260924 07:13:54.043393 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.043039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:57062)
0924 07:13:54.043210 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.043274 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 07:13:54.043548 14153 catalog_manager.cc:4584] TS b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9e5c3d57f4042edbab88e60279f28bc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection refused (error 111)
I20260924 07:13:54.047300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14497
I20260924 07:13:54.048751 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.048458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.0.1:52718)
0924 07:13:54.048593 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.048652 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:13:54.052358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14135
I20260924 07:13:54.058209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--webserver_interface=127.0.98.62
--webserver_port=38121
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:54.172945 14670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:54.173243 14670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:54.173283 14670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:54.177505 14670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:54.177587 14670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:54.177606 14670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:54.177623 14670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:54.177637 14670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:54.183218 14670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:54.184388 14670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:54.185452 14670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:13:54.198714 14670 server_base.cc:1034] running on GCE node
W20260924 07:13:54.198892 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.198936 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.198928 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.199676 14670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:54.200891 14670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:54.202056 14670 hybrid_clock.cc:648] HybridClock initialized: now 1790234034202024 us; error 50 us; skew 500 ppm
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:54.204972 14670 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:54.205940 14670 webserver.cc:460] Webserver started at http://127.0.98.62:38121/ using document root <none> and password file <none>
I20260924 07:13:54.206516 14670 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:54.206598 14670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:54.210798 14670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:13:54.213384 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.214972 14670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:54.215116 14670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "d9e79f91e38849e18419ab355fbb14b4"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-cnv3"
server_key: "a9e702b2ac186f26fa923fe1cdc4f3b6"
server_key_iv: "ea27f3b2b6d28b574a6fa92eaaf3765d"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.215870 14670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:54.227115 14670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:54.230937 14670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:54.231150 14670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:54.238564 14670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41549
I20260924 07:13:54.238821 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41549 every 8 connection(s)
I20260924 07:13:54.239785 14670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:54.245083 14739 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:13:54.245159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14670
I20260924 07:13:54.245594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37647
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=43059
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:54.247123 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap starting.
I20260924 07:13:54.264580 14739 log.cc:824] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:54.273626 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:54.274040 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap complete.
I20260924 07:13:54.278028 14739 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:54.278432 14739 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.278930 14739 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:54.279062 14739 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:54.279131 14739 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:54.279225 14739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:13:54.281301 14739 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:54.281646 14739 leader_election.cc:302] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e79f91e38849e18419ab355fbb14b4; no voters:
I20260924 07:13:54.281965 14739 leader_election.cc:288] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:13:54.282099 14743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:13:54.282308 14743 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 LEADER]: Becoming Leader. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Running, Role: LEADER
I20260924 07:13:54.282634 14743 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:54.283408 14739 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:54.284080 14744 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:54.284216 14744 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:54.284919 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e79f91e38849e18419ab355fbb14b4. Latest consensus state: current_term: 2 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:54.285074 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:54.285297 14750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:54.290540 14750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:54.291253 14750 catalog_manager.cc:1259] Loaded cluster ID: 2a9ae31dfeb346f8be70f7bf15b7ee3b
I20260924 07:13:54.291317 14750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:54.292706 14750 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 07:13:54.297132 14750 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 07:13:54.303099 14750 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:54.304080 14750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Loaded TSK: 0
I20260924 07:13:54.305382 14750 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:13:54.368970 14741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:54.369240 14741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:54.369284 14741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:54.373631 14741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:54.373705 14741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:54.373767 14741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:54.379228 14741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43059
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:54.380450 14741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:54.381579 14741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:54.391844 14767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.391815 14769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.391899 14766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.392390 14741 server_base.cc:1034] running on GCE node
I20260924 07:13:54.392812 14741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:54.393570 14741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:54.394779 14741 hybrid_clock.cc:648] HybridClock initialized: now 1790234034394757 us; error 37 us; skew 500 ppm
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:54.398074 14741 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:54.399225 14741 webserver.cc:460] Webserver started at http://127.0.98.1:43059/ using document root <none> and password file <none>
I20260924 07:13:54.399879 14741 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:54.399977 14741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:54.404376 14741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:13:54.406731 14776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.408171 14741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:54.408290 14741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "b73858d5e3ce4fafb4257faa98771f7c"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "f17c507c49534d5f6e0b019b90aa5350"
server_key_iv: "d3cb48a084213991019e475ac2c41a9c"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.408912 14741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:54.422086 14741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:54.425801 14741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:54.425997 14741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:54.426856 14741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:54.428201 14783 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:13:54.432710 14741 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:13:54.432799 14741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 07:13:54.432857 14741 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:13:54.434849 14741 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:13:54.434909 14741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:13:54.435267 14783 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap starting.
I20260924 07:13:54.444787 14741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37647
I20260924 07:13:54.444906 14890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37647 every 8 connection(s)
I20260924 07:13:54.445801 14741 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:54.452324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14741
I20260924 07:13:54.452733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41473
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=44975
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:54.453986 14783 log.cc:824] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:54.462589 14783 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:54.463052 14783 tablet_bootstrap.cc:492] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap complete.
I20260924 07:13:54.464146 14783 ts_tablet_manager.cc:1397] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.011s
I20260924 07:13:54.465715 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.447937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52863 (local address 127.0.98.62:41549)
0924 07:13:54.448370 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.448395 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.448710 (+ 315us) server_negotiation.cc:407] Connection header received
0924 07:13:54.449929 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.449947 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.450504 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.450924 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.452162 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.453636 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.455278 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.456014 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.456195 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.459508 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.459517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.459538 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:54.459582 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.461849 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.462546 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.462553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.462559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.462741 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.463199 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.463203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.463205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.463721 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.463851 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.464532 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.464848 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
I20260924 07:13:54.465723 14894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.447842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:52863)
0924 07:13:54.448513 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.448572 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.449431 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.450982 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.450989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.451403 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:54.451940 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.451955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.453827 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.453831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.455133 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.455145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.456187 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.456191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.456419 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.457151 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:54.457174 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:54.459192 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:54.462008 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.462015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.462029 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.462409 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.462851 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.462855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.462857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.463028 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.463852 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:54.463858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:54.464178 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.464379 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.464738 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":141,"threads_started":1}
I20260924 07:13:54.467176 14783 raft_consensus.cc:348] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:54.467736 14783 raft_consensus.cc:729] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73858d5e3ce4fafb4257faa98771f7c, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.468230 14783 consensus_queue.cc:260] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } } peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } }
I20260924 07:13:54.468917 14891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:54.469159 14891 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:54.469290 14783 ts_tablet_manager.cc:1428] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:13:54.469740 14891 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:54.472823 14703 ts_manager.cc:194] Registered new tserver with Master: b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647)
I20260924 07:13:54.477422 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52863
I20260924 07:13:54.480436 14891 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:13:54.487862 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.476904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:57070)
0924 07:13:54.477747 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.477795 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.477953 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.478472 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.478478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.478857 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:54.478981 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.478989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.480065 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.480068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.480795 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.480806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.481562 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.481580 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.481701 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.482328 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:54.482344 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:54.483881 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:54.485809 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.485820 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.485826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.486130 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.486458 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.486461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.486463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.486559 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.487128 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:54.487134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:54.487271 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.487461 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.487679 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":146,"threads_started":1}
I20260924 07:13:54.487898 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.476985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.0.98.1:37647)
0924 07:13:54.477483 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.477495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.477929 (+ 434us) server_negotiation.cc:407] Connection header received
0924 07:13:54.478111 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.478115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.478323 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.478462 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.479108 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.479948 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.480928 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.481444 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.481553 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.484031 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.484041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.484051 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:54.484075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.485689 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.486267 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.486273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.486275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.486362 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.486677 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.486680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.486681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.487016 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.487138 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.487608 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.487762 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":123,"threads_started":1}
I20260924 07:13:54.488951 14825 tablet_service.cc:1514] Processing DeleteTablet for tablet f9e5c3d57f4042edbab88e60279f28bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57070
I20260924 07:13:54.489849 14900 tablet_replica.cc:330] stopping tablet replica
I20260924 07:13:54.490182 14900 raft_consensus.cc:2229] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:54.490460 14900 raft_consensus.cc:2258] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:54.491712 14900 ts_tablet_manager.cc:1905] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:54.496605 14900 ts_tablet_manager.cc:1918] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:54.496737 14900 log.cc:1196] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/f9e5c3d57f4042edbab88e60279f28bc
I20260924 07:13:54.497040 14900 ts_tablet_manager.cc:1939] T f9e5c3d57f4042edbab88e60279f28bc P b73858d5e3ce4fafb4257faa98771f7c: Deleting consensus metadata
I20260924 07:13:54.499282 14689 catalog_manager.cc:4841] TS b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647): tablet f9e5c3d57f4042edbab88e60279f28bc (table test-table [id=54720e24c40f4e668013b33b0ebd5dbb]) successfully deleted
W20260924 07:13:54.578513 14895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:54.578805 14895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:54.578852 14895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:54.583017 14895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:54.583102 14895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:54.583165 14895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:54.589105 14895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44975
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:54.590206 14895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:54.591226 14895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:54.601353 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.601353 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.601342 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.601591 14895 server_base.cc:1034] running on GCE node
I20260924 07:13:54.602157 14895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:54.602830 14895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:54.603995 14895 hybrid_clock.cc:648] HybridClock initialized: now 1790234034603986 us; error 30 us; skew 500 ppm
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:54.606758 14895 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:54.607765 14895 webserver.cc:460] Webserver started at http://127.0.98.2:44975/ using document root <none> and password file <none>
I20260924 07:13:54.608345 14895 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:54.608440 14895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:54.612476 14895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:13:54.614643 14915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.616011 14895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:54.616127 14895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "4828c87e53604025ef055e5daf32c625"
server_key_iv: "54e25747a4566300d70d6f393ac575b7"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.616673 14895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:54.643574 14895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:54.647125 14895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:54.647321 14895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:54.647950 14895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:54.649066 14922 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 07:13:54.653244 14922 ts_tablet_manager.cc:1852] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 07:13:54.653672 14922 ts_tablet_manager.cc:1905] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:54.657836 14922 ts_tablet_manager.cc:1918] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:54.658022 14922 ts_tablet_manager.cc:1939] T f9e5c3d57f4042edbab88e60279f28bc P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting consensus metadata
I20260924 07:13:54.658545 14895 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 07:13:54.658622 14895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260924 07:13:54.658697 14895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:54.658730 14895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.668589 14895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41473
I20260924 07:13:54.668666 15029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41473 every 8 connection(s)
I20260924 07:13:54.669534 14895 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:54.670048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 14895
I20260924 07:13:54.670475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42747
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=40105
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:54.684238 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.671796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55297 (local address 127.0.98.62:41549)
0924 07:13:54.671958 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.671963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.672970 (+ 1007us) server_negotiation.cc:407] Connection header received
0924 07:13:54.673980 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.673984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.674065 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.674167 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.675487 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.676016 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.677211 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.677451 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.677543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.680601 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.680605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.680607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:54.680638 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.682006 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.682548 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.682551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.682552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.682635 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.682985 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.682989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.682991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.683185 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.683304 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.683891 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.684060 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:13:54.685243 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.672231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:55297)
0924 07:13:54.672774 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.672834 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.673753 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.674357 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.674366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.674742 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:54.675278 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.675294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.676158 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.676161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.677077 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.677096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.677618 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.677622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.677795 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.678507 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:54.678524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:54.680331 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:54.682133 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.682141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.682161 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.682440 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.682724 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.682727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.682730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.682892 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.683324 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:54.683331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:54.683646 (+ 315us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.683831 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.684365 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260924 07:13:54.686625 15030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:54.686920 15030 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:54.687484 15030 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:54.688676 14703 ts_manager.cc:194] Registered new tserver with Master: abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
I20260924 07:13:54.689268 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55297
W20260924 07:13:54.785537 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:54.785866 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:54.785933 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:54.790071 15032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:54.790134 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:54.790216 15032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:54.795745 15032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40105
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:54.796837 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:54.797899 15032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:54.807659 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.807669 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.807698 15032 server_base.cc:1034] running on GCE node
W20260924 07:13:54.807659 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.808396 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:54.809070 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:54.810246 15032 hybrid_clock.cc:648] HybridClock initialized: now 1790234034810209 us; error 53 us; skew 500 ppm
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:54.813182 15032 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:54.814091 15032 webserver.cc:460] Webserver started at http://127.0.98.3:40105/ using document root <none> and password file <none>
I20260924 07:13:54.814635 15032 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:54.814705 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:54.818768 15032 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:54.821076 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.822367 15032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:13:54.822464 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "bd2eaf3272a944aeae87aa918b61de75"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "e394180ec6000af28033307113ecd3bc"
server_key_iv: "29270bd7768be66f85b496ff35275f74"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.823004 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:54.836364 15032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:54.839627 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:54.839787 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:54.840340 15032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:54.841193 15032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:54.841261 15032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.841305 15032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:54.841318 15032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.850049 15032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42747
I20260924 07:13:54.850083 15161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42747 every 8 connection(s)
I20260924 07:13:54.851099 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:54.857358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15032
Sep 24 07:13:54 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:54.865833 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.858094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.0.98.62:41549)
0924 07:13:54.858336 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.858340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.858484 (+ 144us) server_negotiation.cc:407] Connection header received
0924 07:13:54.858610 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.858614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.858655 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.858725 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.859457 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.859974 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.860824 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.861052 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.861139 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.862625 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.862629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.862631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:54.862657 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.864215 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.864621 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.864625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.864627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.864702 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.864942 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.864946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.864948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.865140 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.865350 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.865508 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.865647 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260924 07:13:54.866779 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.853292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52873 (local address 127.0.98.62:41549)
0924 07:13:54.853433 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.853436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.854610 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 07:13:54.856029 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.856032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.856087 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.856176 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.857697 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.858443 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.859724 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.859958 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.860048 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.862851 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.862855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.862857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:54.862896 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.864324 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.864880 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.864883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.864885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.864967 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:54.865373 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:54.865376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:54.865378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:54.865635 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.865846 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.866504 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.866629 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":54}
I20260924 07:13:54.868005 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.853874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:52873)
0924 07:13:54.854441 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.854488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.855754 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.856359 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.856367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.856813 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:54.857485 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.857507 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.858586 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.858589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.859607 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.859617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.860126 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.860130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.860300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.860935 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:54.860954 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:54.862669 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:54.864461 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.864469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.864491 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.864781 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.865057 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:54.865060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:54.865063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:54.865218 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:54.865846 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:54.865853 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:54.866250 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.866446 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.866844 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":148,"threads_started":1}
I20260924 07:13:54.869136 15162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41549
I20260924 07:13:54.869400 15162 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:54.869997 15162 heartbeater.cc:511] Master 127.0.98.62:41549 requested a full tablet report, sending...
I20260924 07:13:54.871138 14703 ts_manager.cc:194] Registered new tserver with Master: bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747)
I20260924 07:13:54.871821 14703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52873
I20260924 07:13:54.880007 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:54.884676 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.880691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.0.98.62:41549)
0924 07:13:54.880827 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.880832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.881029 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:13:54.881205 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.881207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.881323 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.881419 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:54.881694 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.882208 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.883089 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.883312 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.883435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.883800 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:54.884236 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:54.884351 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.884444 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.884508 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:13:54.885397 14703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:54.888878 14703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:54.897054 14825 tablet_service.cc:1467] Processing CreateTablet for tablet 13fc529a00d545d595c44c67c12298ff (DEFAULT_TABLE table=test-table [id=fada866a17af4209a9fdfafce84c77ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:54.897764 14825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fc529a00d545d595c44c67c12298ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:54.900808 14783 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap starting.
I20260924 07:13:54.901495 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.895757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42747 (local address 127.0.0.1:46868)
0924 07:13:54.895926 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.895938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.896018 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.897835 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.897838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.897854 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:54.897946 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.897953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.899181 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.899184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.899977 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.899987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.901023 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.901027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.901172 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.901217 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.901270 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.901353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1}
I20260924 07:13:54.901779 14783 tablet_bootstrap.cc:654] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:54.902016 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.896066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.0.1:52732)
0924 07:13:54.896532 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:54.896545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:54.896632 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:54.897144 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:54.897146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:54.897165 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:54.897272 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.897279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.898684 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.898688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.899921 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:54.899932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.901008 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.901011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:54.901098 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.901274 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 07:13:54.901348 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:13:54.901423 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":132,"thread_start_us":120,"threads_started":1}
I20260924 07:13:54.902204 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.896122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46868 (local address 127.0.98.3:42747)
0924 07:13:54.897412 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.897424 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.897447 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:13:54.897524 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.897530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.897693 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.897834 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:54.898067 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.899051 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.900093 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.900899 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.901765 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.901917 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.901966 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.902057 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":103,"threads_started":1}
I20260924 07:13:54.902354 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.896349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.0.98.2:41473)
0924 07:13:54.896704 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.896714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.896736 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:13:54.896803 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.896807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.896995 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.897127 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:54.897416 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.898539 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.900057 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.900662 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.900778 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.900940 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.901779 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.902031 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
I20260924 07:13:54.903441 14783 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: No bootstrap required, opened a new log
I20260924 07:13:54.903543 14783 ts_tablet_manager.cc:1397] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260924 07:13:54.904021 14783 raft_consensus.cc:348] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.904142 14783 raft_consensus.cc:374] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:54.904176 14783 raft_consensus.cc:729] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73858d5e3ce4fafb4257faa98771f7c, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.903837 14964 tablet_service.cc:1467] Processing CreateTablet for tablet 13fc529a00d545d595c44c67c12298ff (DEFAULT_TABLE table=test-table [id=fada866a17af4209a9fdfafce84c77ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:54.904336 14783 consensus_queue.cc:260] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.904677 14783 ts_tablet_manager.cc:1428] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:54.904982 14964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fc529a00d545d595c44c67c12298ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:54.905165 15096 tablet_service.cc:1467] Processing CreateTablet for tablet 13fc529a00d545d595c44c67c12298ff (DEFAULT_TABLE table=test-table [id=fada866a17af4209a9fdfafce84c77ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:54.906160 15096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fc529a00d545d595c44c67c12298ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:54.909572 14922 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Bootstrap starting.
I20260924 07:13:54.912160 14922 tablet_bootstrap.cc:654] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:54.913086 14922 log.cc:824] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:54.914069 15176 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Bootstrap starting.
I20260924 07:13:54.915352 14922 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: No bootstrap required, opened a new log
I20260924 07:13:54.915585 14922 ts_tablet_manager.cc:1397] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260924 07:13:54.916293 15176 tablet_bootstrap.cc:654] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:54.917320 15176 log.cc:824] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:54.918529 14922 raft_consensus.cc:348] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.918752 14922 raft_consensus.cc:374] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:54.918787 14922 raft_consensus.cc:729] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc78d1c6cbf49e4b7c60b54e7fc2167, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.919123 15176 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: No bootstrap required, opened a new log
I20260924 07:13:54.919232 14922 consensus_queue.cc:260] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.919317 15176 ts_tablet_manager.cc:1397] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:13:54.920038 15030 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
I20260924 07:13:54.920239 14922 ts_tablet_manager.cc:1428] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
W20260924 07:13:54.920543 15031 tablet.cc:2369] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:54.922993 15176 raft_consensus.cc:348] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.923225 15176 raft_consensus.cc:374] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:54.923280 15176 raft_consensus.cc:729] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2eaf3272a944aeae87aa918b61de75, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.923738 15176 consensus_queue.cc:260] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:54.924413 15162 heartbeater.cc:503] Master 127.0.98.62:41549 was elected leader, sending a full tablet report...
I20260924 07:13:54.924639 15176 ts_tablet_manager.cc:1428] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 07:13:54.947199 14892 tablet.cc:2369] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:55.058440 15179 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:55.058683 15179 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:55.060040 15179 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473), b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647)
I20260924 07:13:55.063957 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.060475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50255 (local address 127.0.98.2:41473)
0924 07:13:55.060617 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:55.060622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:55.060948 (+ 326us) server_negotiation.cc:407] Connection header received
0924 07:13:55.061060 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:55.061062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:55.061107 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:55.061168 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:55.061420 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.061900 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.062922 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.063327 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.063477 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:55.063548 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:55.063755 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 07:13:55.063830 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:13:55.064007 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.060348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.98.3:50255)
0924 07:13:55.060846 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.060856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:55.060958 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:55.061189 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:55.061191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:55.061206 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:55.061313 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:55.061318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.062000 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.062004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:55.062803 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:55.062812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.063487 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.063491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:55.063609 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:55.063663 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 07:13:55.063729 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:13:55.063807 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":185,"thread_start_us":118,"threads_started":1}
I20260924 07:13:55.064067 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.060348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.98.3:48481)
0924 07:13:55.060550 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.060569 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:55.060680 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:55.061042 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:55.061045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:55.061063 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:55.061172 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:55.061178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.061918 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.061921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:55.062745 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:55.062755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.063578 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.063582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:55.063742 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:55.063763 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:13:55.063847 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:13:55.063924 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":46}
I20260924 07:13:55.064260 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.060475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48481 (local address 127.0.98.1:37647)
0924 07:13:55.060761 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:55.060764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:55.060776 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:13:55.060826 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:55.060829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:55.060880 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:55.060999 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:55.061292 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.061806 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.062881 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.063365 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.063596 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:55.063782 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:55.063862 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:13:55.063938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":115,"threads_started":1}
I20260924 07:13:55.064638 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fc529a00d545d595c44c67c12298ff" candidate_uuid: "bd2eaf3272a944aeae87aa918b61de75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" is_pre_election: true
I20260924 07:13:55.064786 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fc529a00d545d595c44c67c12298ff" candidate_uuid: "bd2eaf3272a944aeae87aa918b61de75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73858d5e3ce4fafb4257faa98771f7c" is_pre_election: true
I20260924 07:13:55.064991 14984 raft_consensus.cc:2454] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd2eaf3272a944aeae87aa918b61de75 in term 0.
I20260924 07:13:55.065034 14844 raft_consensus.cc:2454] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd2eaf3272a944aeae87aa918b61de75 in term 0.
I20260924 07:13:55.065503 15051 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c, bd2eaf3272a944aeae87aa918b61de75; no voters:
I20260924 07:13:55.065840 15179 raft_consensus.cc:2781] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:55.065927 15179 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:55.065968 15179 raft_consensus.cc:3037] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:55.066992 15179 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:55.067340 15179 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [CANDIDATE]: Term 1 election: Requested vote from peers abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473), b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647)
I20260924 07:13:55.067788 14844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fc529a00d545d595c44c67c12298ff" candidate_uuid: "bd2eaf3272a944aeae87aa918b61de75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b73858d5e3ce4fafb4257faa98771f7c"
I20260924 07:13:55.067788 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13fc529a00d545d595c44c67c12298ff" candidate_uuid: "bd2eaf3272a944aeae87aa918b61de75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
I20260924 07:13:55.067938 14984 raft_consensus.cc:3037] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:55.067938 14844 raft_consensus.cc:3037] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:55.069231 14844 raft_consensus.cc:2454] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2eaf3272a944aeae87aa918b61de75 in term 1.
I20260924 07:13:55.069231 14984 raft_consensus.cc:2454] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2eaf3272a944aeae87aa918b61de75 in term 1.
I20260924 07:13:55.069569 15051 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c, bd2eaf3272a944aeae87aa918b61de75; no voters:
I20260924 07:13:55.069748 15179 raft_consensus.cc:2781] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:55.069990 15179 raft_consensus.cc:686] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 1 LEADER]: Becoming Leader. State: Replica: bd2eaf3272a944aeae87aa918b61de75, State: Running, Role: LEADER
I20260924 07:13:55.070344 15179 consensus_queue.cc:237] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:55.073645 14702 catalog_manager.cc:5495] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bd2eaf3272a944aeae87aa918b61de75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:55.096220 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.092831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46880 (local address 127.0.98.3:42747)
0924 07:13:55.092950 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:55.092954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:55.092967 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:13:55.093044 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:55.093046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:55.093130 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:55.093219 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:55.093405 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.093897 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.094732 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:55.094956 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:55.095037 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:55.095303 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:55.095794 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:55.095888 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:55.096002 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:13:55.096061 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:13:55.102272 15163 tablet.cc:2369] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:55.102448 14844 raft_consensus.cc:1261] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Refusing update from remote peer bd2eaf3272a944aeae87aa918b61de75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:55.102448 14984 raft_consensus.cc:1261] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Refusing update from remote peer bd2eaf3272a944aeae87aa918b61de75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:55.103222 15182 consensus_queue.cc:1035] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:55.103437 15179 consensus_queue.cc:1035] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:55.110848 15187 mvcc.cc:205] Tried to move back new op lower bound from 7332798607771676672 to 7332798607650947072. Current Snapshot: MvccSnapshot[applied={T|T < 7332798607771676672}]
I20260924 07:13:55.111174 15186 mvcc.cc:205] Tried to move back new op lower bound from 7332798607771676672 to 7332798607650947072. Current Snapshot: MvccSnapshot[applied={T|T < 7332798607771676672}]
I20260924 07:13:55.119426 14703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:55.119644 14703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:55.122179 14703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Sending DeleteTablet for 3 replicas of tablet 13fc529a00d545d595c44c67c12298ff
I20260924 07:13:55.122964 14964 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fc529a00d545d595c44c67c12298ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52732
I20260924 07:13:55.122977 15096 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fc529a00d545d595c44c67c12298ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46868
I20260924 07:13:55.123461 15195 tablet_replica.cc:330] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: stopping tablet replica
I20260924 07:13:55.123665 14825 tablet_service.cc:1514] Processing DeleteTablet for tablet 13fc529a00d545d595c44c67c12298ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57070
I20260924 07:13:55.123786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14741
I20260924 07:13:55.123827 15195 raft_consensus.cc:2229] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:55.124184 15195 raft_consensus.cc:2258] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:55.125686 15195 ts_tablet_manager.cc:1905] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:55.127588 15196 tablet_replica.cc:330] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: stopping tablet replica
I20260924 07:13:55.128015 15196 raft_consensus.cc:2229] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:55.128576 15196 raft_consensus.cc:2258] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:55.128650 15195 ts_tablet_manager.cc:1918] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:55.128755 15195 log.cc:1196] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/13fc529a00d545d595c44c67c12298ff
I20260924 07:13:55.129048 15195 ts_tablet_manager.cc:1939] T 13fc529a00d545d595c44c67c12298ff P abc78d1c6cbf49e4b7c60b54e7fc2167: Deleting consensus metadata
I20260924 07:13:55.130438 14690 catalog_manager.cc:4841] TS abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): tablet 13fc529a00d545d595c44c67c12298ff (table test-table [id=fada866a17af4209a9fdfafce84c77ff]) successfully deleted
W20260924 07:13:55.131810 14689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37647 (error 108)
W20260924 07:13:55.133106 15171 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:55.132798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37647 (local address 127.0.0.1:57080)
0924 07:13:55.132938 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.133008 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 07:13:55.133282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14895
I20260924 07:13:55.133291 15196 ts_tablet_manager.cc:1905] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:13:55.133285 14689 catalog_manager.cc:4584] TS b73858d5e3ce4fafb4257faa98771f7c (127.0.98.1:37647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13fc529a00d545d595c44c67c12298ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37647: connect: Connection reset by peer (error 104)
I20260924 07:13:55.136250 15196 ts_tablet_manager.cc:1918] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:55.136426 15196 log.cc:1196] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/13fc529a00d545d595c44c67c12298ff
I20260924 07:13:55.136825 15196 ts_tablet_manager.cc:1939] T 13fc529a00d545d595c44c67c12298ff P bd2eaf3272a944aeae87aa918b61de75: Deleting consensus metadata
I20260924 07:13:55.138227 14689 catalog_manager.cc:4841] TS bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): tablet 13fc529a00d545d595c44c67c12298ff (table test-table [id=fada866a17af4209a9fdfafce84c77ff]) successfully deleted
I20260924 07:13:55.139662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15032
I20260924 07:13:55.144964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 14670
I20260924 07:13:55.150826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--webserver_interface=127.0.98.62
--webserver_port=38121
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:55.266674 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:55.266974 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:55.267035 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:55.271767 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:55.271857 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:55.271885 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:55.271909 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:55.271929 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:55.277683 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:55.278863 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:55.279999 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:55.288925 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.288937 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.288977 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.289278 15198 server_base.cc:1034] running on GCE node
I20260924 07:13:55.289945 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:55.291095 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:55.292272 15198 hybrid_clock.cc:648] HybridClock initialized: now 1790234035292238 us; error 46 us; skew 500 ppm
Sep 24 07:13:55 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:55.295499 15198 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:55.296571 15198 webserver.cc:460] Webserver started at http://127.0.98.62:38121/ using document root <none> and password file <none>
I20260924 07:13:55.297144 15198 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:55.297207 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:55.301785 15198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:13:55.304220 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.305909 15198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:55.306062 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "d9e79f91e38849e18419ab355fbb14b4"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-cnv3"
server_key: "a9e702b2ac186f26fa923fe1cdc4f3b6"
server_key_iv: "ea27f3b2b6d28b574a6fa92eaaf3765d"
server_key_version: "encryptionkey@0"
I20260924 07:13:55.306684 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:55.333248 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:55.336892 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:55.337107 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:55.344624 15198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41549
I20260924 07:13:55.344895 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41549 every 8 connection(s)
I20260924 07:13:55.345793 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:55.347572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15198
I20260924 07:13:55.348038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37647
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=43059
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:55.351055 15266 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:13:55.352937 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap starting.
I20260924 07:13:55.370899 15266 log.cc:824] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:55.386045 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:55.386516 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4: Bootstrap complete.
I20260924 07:13:55.392076 15266 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:55.392625 15266 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Initialized, Role: FOLLOWER
I20260924 07:13:55.393215 15266 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:55.393389 15266 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:55.393472 15266 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:55.393584 15266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:13:55.396304 15266 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:55.396705 15266 leader_election.cc:302] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9e79f91e38849e18419ab355fbb14b4; no voters:
I20260924 07:13:55.397073 15266 leader_election.cc:288] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:13:55.397219 15270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:13:55.397487 15270 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [term 3 LEADER]: Becoming Leader. State: Replica: d9e79f91e38849e18419ab355fbb14b4, State: Running, Role: LEADER
I20260924 07:13:55.397836 15270 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } }
I20260924 07:13:55.398414 15266 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:55.399353 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:55.399559 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:55.399391 15272 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9e79f91e38849e18419ab355fbb14b4. Latest consensus state: current_term: 3 leader_uuid: "d9e79f91e38849e18419ab355fbb14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9e79f91e38849e18419ab355fbb14b4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41549 } } }
I20260924 07:13:55.399780 15272 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9e79f91e38849e18419ab355fbb14b4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:55.401192 15274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:55.406216 15274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:55.406893 15274 catalog_manager.cc:1259] Loaded cluster ID: 2a9ae31dfeb346f8be70f7bf15b7ee3b
I20260924 07:13:55.406944 15274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 07:13:55.408094 15274 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f44b89f69ad google::LogMessage::Fail() at ??:0
@ 0x7f44b89f988a google::LogMessage::SendToLog() at ??:0
@ 0x7f44b89f64de google::LogMessage::Flush() at ??:0
@ 0x7f44b89f7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f44c3335fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f44c3336d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f44c3333579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f44c3377640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f44bac483d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f44bac48c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f44bac4a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f44bac39512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f44bc27d6db start_thread at ??:0
@ 0x7f44b74f461f clone at ??:0
W20260924 07:13:55.491055 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:55.491353 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:55.491438 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:55.495697 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:55.495771 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:55.495855 15268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:55.501533 15268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43059
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:55.502679 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:55.503841 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:55.514146 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.514299 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.514146 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.514283 15268 server_base.cc:1034] running on GCE node
I20260924 07:13:55.514997 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:55.515720 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:55.516893 15268 hybrid_clock.cc:648] HybridClock initialized: now 1790234035516852 us; error 58 us; skew 500 ppm
Sep 24 07:13:55 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:55.519786 15268 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:55.520744 15268 webserver.cc:460] Webserver started at http://127.0.98.1:43059/ using document root <none> and password file <none>
I20260924 07:13:55.521375 15268 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:55.521456 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:55.525583 15268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:13:55.527907 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.529413 15268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:55.529565 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "b73858d5e3ce4fafb4257faa98771f7c"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "f17c507c49534d5f6e0b019b90aa5350"
server_key_iv: "d3cb48a084213991019e475ac2c41a9c"
server_key_version: "encryptionkey@0"
I20260924 07:13:55.530350 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:55.560899 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:55.565227 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:55.565444 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:55.566200 15268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:55.567669 15324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:13:55.572000 15268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:13:55.572103 15268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 07:13:55.572192 15268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:13:55.574786 15268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:13:55.574864 15268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 07:13:55.575228 15324 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap starting.
I20260924 07:13:55.585588 15268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37647
I20260924 07:13:55.585611 15431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37647 every 8 connection(s)
I20260924 07:13:55.586859 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:55.594331 15324 log.cc:824] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:55.595747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15268
I20260924 07:13:55.596190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41473
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=44975
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:55.602687 15324 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:55.603118 15324 tablet_bootstrap.cc:492] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Bootstrap complete.
I20260924 07:13:55.604414 15324 ts_tablet_manager.cc:1397] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.008s
I20260924 07:13:55.607956 15324 raft_consensus.cc:348] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:55.608563 15324 raft_consensus.cc:729] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b73858d5e3ce4fafb4257faa98771f7c, State: Initialized, Role: FOLLOWER
I20260924 07:13:55.609162 15324 consensus_queue.cc:260] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:55.610339 15324 ts_tablet_manager.cc:1428] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260924 07:13:55.642030 15434 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:55.589229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:60433)
0924 07:13:55.589861 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.589927 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:55.591035 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:55.641271 (+ 50236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":186,"threads_started":1}
W20260924 07:13:55.642525 15432 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41549: Client connection negotiation failed: client connection to 127.0.98.62:41549: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:13:55.643277 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.643037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:57019)
0924 07:13:55.643154 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.643218 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:13:55.643898 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.643716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:51265)
0924 07:13:55.643808 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.643850 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:13:55.644057 15432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:13:55.716746 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:55.717036 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:55.717083 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:55.721379 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:55.721455 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:55.721529 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:55.727025 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44975
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:55.728135 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:55.729156 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:55.739413 15443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.739477 15436 server_base.cc:1034] running on GCE node
W20260924 07:13:55.739425 15445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.739424 15442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.740064 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:55.740727 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:55.741979 15436 hybrid_clock.cc:648] HybridClock initialized: now 1790234035741957 us; error 34 us; skew 500 ppm
Sep 24 07:13:55 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:55.744890 15436 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:55.745905 15436 webserver.cc:460] Webserver started at http://127.0.98.2:44975/ using document root <none> and password file <none>
I20260924 07:13:55.746490 15436 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:55.746564 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:55.751029 15436 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:55.753633 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.755132 15436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:13:55.755285 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "4828c87e53604025ef055e5daf32c625"
server_key_iv: "54e25747a4566300d70d6f393ac575b7"
server_key_version: "encryptionkey@0"
I20260924 07:13:55.755954 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:55.775007 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:55.778419 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:55.778612 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:55.779237 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:55.780122 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:55.780181 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.780238 15436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:55.780264 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.789291 15436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41473
I20260924 07:13:55.789324 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41473 every 8 connection(s)
I20260924 07:13:55.790321 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:55.794399 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.792546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:51893)
0924 07:13:55.793052 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.793179 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":124,"threads_started":1}
W20260924 07:13:55.794925 15566 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41549: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
I20260924 07:13:55.795622 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.795307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:48437)
0924 07:13:55.795454 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.795563 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:13:55.796223 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.796034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:42177)
0924 07:13:55.796134 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.796176 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:13:55.796387 15566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:13:55.798228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15436
I20260924 07:13:55.798645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42747
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41549
--webserver_port=40105
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:55.912999 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:55.913305 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:55.913349 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:55.917481 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:55.917555 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:55.917618 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:55.923272 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40105
--tserver_master_addrs=127.0.98.62:41549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:13:55.924398 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:55.925405 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:55.935490 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:55.935875 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.935920 15569 server_base.cc:1034] running on GCE node
W20260924 07:13:55.935993 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:55.936353 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:55.937040 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:55.938244 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790234035938224 us; error 35 us; skew 500 ppm
Sep 24 07:13:55 dist-test-slave-cnv3 krb5kdc[14122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:55.941259 15569 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:55.942310 15569 webserver.cc:460] Webserver started at http://127.0.98.3:40105/ using document root <none> and password file <none>
I20260924 07:13:55.942898 15569 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:55.942967 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:55.947167 15569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:55.949470 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.951016 15569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:55.951150 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "bd2eaf3272a944aeae87aa918b61de75"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-cnv3"
server_key: "e394180ec6000af28033307113ecd3bc"
server_key_iv: "29270bd7768be66f85b496ff35275f74"
server_key_version: "encryptionkey@0"
I20260924 07:13:55.951808 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:55.967446 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:55.970706 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:55.970891 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:55.971518 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:55.972369 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:55.972417 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.972472 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:55.972500 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:55.981171 15569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42747
I20260924 07:13:55.981194 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42747 every 8 connection(s)
I20260924 07:13:55.982228 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:55.985373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15569
I20260924 07:13:55.986335 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.984459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:45463)
0924 07:13:55.984963 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.985080 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
W20260924 07:13:55.986850 15698 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41549 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41549: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
I20260924 07:13:55.987452 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.987187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:51809)
0924 07:13:55.987319 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.987368 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:13:55.988236 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:55.987984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:49347)
0924 07:13:55.988102 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:55.988179 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 07:13:55.988430 15698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:13:56.645159 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:56.644693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:38907)
0924 07:13:56.644980 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:56.645044 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260924 07:13:56.797526 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:56.797087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:40385)
0924 07:13:56.797355 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:56.797417 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260924 07:13:56.989511 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:56.989048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:51209)
0924 07:13:56.989327 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:56.989395 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260924 07:13:57.341358 15705 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:57.341547 15705 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:57.342769 15705 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
Sep 24 07:13:57 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 07:13:57 dist-test-slave-cnv3 krb5kdc[14122](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 07:13:57.357717 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.343189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45471 (local address 127.0.98.3:42747)
0924 07:13:57.343672 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.343695 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.343742 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:13:57.344470 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.344476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.344836 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.345091 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.346403 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.347982 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349298 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.349804 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349967 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.352853 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.352859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.352875 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.352910 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.354744 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355370 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.355403 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.355409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.355542 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355906 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.355910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.355911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.356500 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.356611 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.357235 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.357508 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":98,"threads_started":1}
I20260924 07:13:57.357893 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.343220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56391 (local address 127.0.98.2:41473)
0924 07:13:57.343600 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.343617 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.343819 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:13:57.344465 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.344471 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.344836 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.345101 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.346414 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.347982 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349313 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.349799 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349963 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.353112 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.353117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.353130 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.353162 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.355079 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355588 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.355594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.355599 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.355767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.356138 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.356141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.356143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.356557 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.356685 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.357414 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.357711 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
I20260924 07:13:57.357920 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.343050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42747 (local address 127.0.98.1:45471)
0924 07:13:57.343425 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.343442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.343555 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.345225 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.345231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.345713 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.346213 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.346229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.348159 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.348162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.349164 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.349176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.349931 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.350149 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.350845 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.350856 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.352579 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.354888 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.354895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.354911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.355213 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.355632 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.355637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.355802 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.356623 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.356630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.356903 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.357079 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.357463 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":218,"spinlock_wait_cycles":23040,"thread_start_us":76,"threads_started":1}
I20260924 07:13:57.358614 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.343055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41473 (local address 127.0.98.1:56391)
0924 07:13:57.343722 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.343732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.343819 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.345464 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.345466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.345737 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.346214 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.346230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.348156 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.348159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.349165 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.349176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.349931 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.349935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.350146 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.350829 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.350847 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.352885 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.355205 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.355211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.355478 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.355906 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.355909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.355911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.356030 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.356671 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.356679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.356902 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.357079 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.358453 (+ 1374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":1681,"spinlock_wait_cycles":21120,"thread_start_us":60,"threads_started":1}
W20260924 07:13:57.361405 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:13:57.361609 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:13:57.361688 15321 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:13:57.361976 15705 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:57.646252 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.646001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:39669)
0924 07:13:57.646125 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.646174 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 07:13:57.798892 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.798411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:58897)
0924 07:13:57.798700 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.798766 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260924 07:13:57.990767 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.990330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:36573)
0924 07:13:57.990613 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.990672 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260924 07:13:58.647686 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.647141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:37507)
0924 07:13:58.647520 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:58.647587 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":81,"threads_started":1}
I20260924 07:13:58.800067 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.799673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:50719)
0924 07:13:58.799908 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:58.799969 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260924 07:13:58.992201 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.991728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:43847)
0924 07:13:58.992014 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:58.992084 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260924 07:13:59.335204 15715 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:59.335338 15715 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:13:59.335783 15715 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
W20260924 07:13:59.336669 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:13:59.336851 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:13:59.336920 15320 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:13:59.337076 15715 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:59.648944 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.648478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:55875)
0924 07:13:59.648782 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:59.648847 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
I20260924 07:13:59.801182 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.800759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:38641)
0924 07:13:59.801011 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:59.801066 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260924 07:13:59.993496 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.993029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:48195)
0924 07:13:59.993337 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:59.993401 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
I20260924 07:14:00.650825 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.650216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:51727)
0924 07:14:00.650617 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.650697 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260924 07:14:00.802721 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.802142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:49961)
0924 07:14:00.802514 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.802593 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":122,"threads_started":1}
I20260924 07:14:00.995136 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.994525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:49007)
0924 07:14:00.994892 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.995003 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260924 07:14:01.652189 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.651723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:43487)
0924 07:14:01.652021 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:01.652083 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260924 07:14:01.804174 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.803682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:50423)
0924 07:14:01.803993 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:01.804059 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260924 07:14:01.996622 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.996094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:42279)
0924 07:14:01.996412 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:01.996515 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260924 07:14:02.273079 15725 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:02.273214 15725 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:14:02.273669 15725 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
W20260924 07:14:02.274483 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:14:02.274601 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:14:02.274659 15321 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:14:02.274783 15725 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:02.653745 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.653192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:45679)
0924 07:14:02.653537 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:02.653614 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":74,"threads_started":1}
I20260924 07:14:02.805588 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.805063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:36381)
0924 07:14:02.805408 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:02.805477 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":74,"threads_started":1}
I20260924 07:14:02.998049 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.997577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:50647)
0924 07:14:02.997887 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:02.997952 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":78,"threads_started":1}
I20260924 07:14:03.655114 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:03.654666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:51881)
0924 07:14:03.654949 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:03.655018 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260924 07:14:03.806773 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:03.806347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:52743)
0924 07:14:03.806617 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:03.806676 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260924 07:14:03.999318 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:03.998905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:54253)
0924 07:14:03.999155 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:03.999216 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260924 07:14:04.052690 15732 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:04.052886 15732 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:14:04.053319 15732 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
W20260924 07:14:04.054236 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:14:04.054391 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:14:04.054474 15320 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:14:04.054677 15732 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:04.656373 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:04.655964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:55135)
0924 07:14:04.656210 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:04.656278 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
I20260924 07:14:04.808112 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:04.807643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:35347)
0924 07:14:04.807926 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:04.808007 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260924 07:14:05.000552 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:05.000085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:57451)
0924 07:14:05.000383 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:05.000449 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260924 07:14:05.657889 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:05.657421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:56239)
0924 07:14:05.657723 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:05.657789 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
I20260924 07:14:05.809410 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:05.808950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:45315)
0924 07:14:05.809230 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:05.809296 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260924 07:14:06.001963 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:06.001456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:58681)
0924 07:14:06.001760 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:06.001843 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260924 07:14:06.659364 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:06.658767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:46229)
0924 07:14:06.659165 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:06.659237 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":80,"threads_started":1}
I20260924 07:14:06.713704 15740 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:06.713856 15740 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:14:06.714345 15740 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
W20260924 07:14:06.715245 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:14:06.715397 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:14:06.715471 15321 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:14:06.715646 15740 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:06.810608 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:06.810180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:50073)
0924 07:14:06.810435 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:06.810492 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":73,"threads_started":1}
I20260924 07:14:07.003300 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:07.002800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:48163)
0924 07:14:07.003135 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:07.003203 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":82,"threads_started":1}
I20260924 07:14:07.660735 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:07.660243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:59967)
0924 07:14:07.660540 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:07.660618 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":66,"threads_started":1}
I20260924 07:14:07.811944 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:07.811457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:40171)
0924 07:14:07.811777 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:07.811844 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":118,"threads_started":1}
I20260924 07:14:08.004582 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:08.004155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:49539)
0924 07:14:08.004426 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:08.004485 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260924 07:14:08.662011 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:08.661528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:44423)
0924 07:14:08.661837 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:08.661912 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":76,"threads_started":1}
I20260924 07:14:08.762660 15747 raft_consensus.cc:482] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:08.762799 15747 raft_consensus.cc:504] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2eaf3272a944aeae87aa918b61de75" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42747 } } peers { permanent_uuid: "abc78d1c6cbf49e4b7c60b54e7fc2167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41473 } } peers { permanent_uuid: "b73858d5e3ce4fafb4257faa98771f7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37647 } }
I20260924 07:14:08.763195 15747 leader_election.cc:288] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747), abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473)
W20260924 07:14:08.764045 15321 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abc78d1c6cbf49e4b7c60b54e7fc2167 (127.0.98.2:41473): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
W20260924 07:14:08.764153 15320 leader_election.cc:341] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2eaf3272a944aeae87aa918b61de75 (127.0.98.3:42747): Not found: Tablet not found: 13fc529a00d545d595c44c67c12298ff
I20260924 07:14:08.764305 15320 leader_election.cc:302] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b73858d5e3ce4fafb4257faa98771f7c; no voters: abc78d1c6cbf49e4b7c60b54e7fc2167, bd2eaf3272a944aeae87aa918b61de75
I20260924 07:14:08.764462 15747 raft_consensus.cc:2726] T 13fc529a00d545d595c44c67c12298ff P b73858d5e3ce4fafb4257faa98771f7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:08.813288 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:08.812834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:35593)
0924 07:14:08.813125 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:08.813189 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
I20260924 07:14:09.005795 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:09.005362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:50411)
0924 07:14:09.005615 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:09.005673 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 07:14:09.663581 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:09.663000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:59397)
0924 07:14:09.663302 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:09.663473 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":75,"threads_started":1}
I20260924 07:14:09.814713 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:09.814230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:46455)
0924 07:14:09.814540 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:09.814606 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260924 07:14:10.007042 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:10.006589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.3:39437)
0924 07:14:10.006857 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:10.006941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260924 07:14:10.665153 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:10.664656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.1:48877)
0924 07:14:10.664984 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:10.665051 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260924 07:14:10.816058 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:10.815649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41549 (local address 127.0.98.2:51639)
0924 07:14:10.815896 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:10.815951 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260924 07:14:10.989987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15268
I20260924 07:14:10.995096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15436
I20260924 07:14:10.999854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15569
2026-09-24T07:14:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19398 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:14:11 dist-test-slave-cnv3 krb5kdc[15757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:14:11 dist-test-slave-cnv3 krb5kdc[15757](info): set up 2 sockets
Sep 24 07:14:11 dist-test-slave-cnv3 krb5kdc[15757](info): commencing operation
krb5kdc: starting...
W20260924 07:14:11.959547 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:14:12.083915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:12.197747 15770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:12.198061 15770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:12.198120 15770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:12.202320 15770 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:14:12.202389 15770 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:14:12.202411 15770 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:14:12.202430 15770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:12.202452 15770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:12.202484 15770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:12.202513 15770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:12.202544 15770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:12.208154 15770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:12.209273 15770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:12.210345 15770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:12.219589 15776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:12.219581 15775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.219717 15770 server_base.cc:1034] running on GCE node
W20260924 07:14:12.219594 15778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.220494 15770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:12.221692 15770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:12.222882 15770 hybrid_clock.cc:648] HybridClock initialized: now 1790234052222851 us; error 43 us; skew 500 ppm
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:12.225821 15770 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:14:12.226820 15770 webserver.cc:460] Webserver started at http://127.0.98.62:44979/ using document root <none> and password file <none>
I20260924 07:14:12.227437 15770 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:12.227514 15770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:12.227727 15770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:12.229893 15770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "8099982d493d43a3aea0638102d4343f"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "97960cce7cdca35541746cf5768b7b7f"
server_key_iv: "bc4f86640c1b09be32c52a3587f759d8"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.230468 15770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "8099982d493d43a3aea0638102d4343f"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "97960cce7cdca35541746cf5768b7b7f"
server_key_iv: "bc4f86640c1b09be32c52a3587f759d8"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.234444 15770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:14:12.237268 15785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.238708 15770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:14:12.238853 15770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "8099982d493d43a3aea0638102d4343f"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "97960cce7cdca35541746cf5768b7b7f"
server_key_iv: "bc4f86640c1b09be32c52a3587f759d8"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.238973 15770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:12.256901 15770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:12.260313 15770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:12.260506 15770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:12.267884 15770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46559
I20260924 07:14:12.268132 15837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46559 every 8 connection(s)
I20260924 07:14:12.268894 15770 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:14:12.271129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15770
I20260924 07:14:12.271241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:14:12.271545 392 external_mini_cluster.cc:1428] Setting key bdbc26e456f6897f6b5e46df5ca15155
I20260924 07:14:12.271862 15838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:12.279062 15838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap starting.
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:12.281641 15838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:12.282550 15838 log.cc:824] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:12.285033 15838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: No bootstrap required, opened a new log
I20260924 07:14:12.286969 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:12.273010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.0.98.62:46559)
0924 07:14:12.273442 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:12.273461 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:12.273504 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:14:12.274560 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:12.274584 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:12.275027 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:12.275438 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:12.276109 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.277573 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.278442 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.279132 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.279291 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:12.281939 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:12.281946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:12.281964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:12.281997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:12.283827 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.284272 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.284278 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.284284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.284424 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.284684 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.284687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.284689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.285227 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:12.285432 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:12.285642 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 07:14:12.285973 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":109,"threads_started":1}
I20260924 07:14:12.289268 15838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:12.289554 15838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:12.289603 15838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Initialized, Role: FOLLOWER
I20260924 07:14:12.289989 15838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:12.290102 15838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:14:12.290148 15838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:14:12.290227 15838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:12.291317 15838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:12.291698 15838 leader_election.cc:302] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8099982d493d43a3aea0638102d4343f; no voters:
I20260924 07:14:12.291977 15838 leader_election.cc:288] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:14:12.292085 15843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:14:12.292325 15843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 LEADER]: Becoming Leader. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Running, Role: LEADER
I20260924 07:14:12.292675 15843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:12.293184 15838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:12.294230 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8099982d493d43a3aea0638102d4343f. Latest consensus state: current_term: 1 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:12.294358 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:12.294727 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:12.294896 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
W20260924 07:14:12.297915 15858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:14:12.297986 15858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:14:12.298058 15852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:14:12.298774 15852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:14:12.304841 15852 catalog_manager.cc:1347] Generated new cluster ID: 8e09f01f8fdc4491ad2b0f284555270d
I20260924 07:14:12.304925 15852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:14:12.360776 15852 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:14:12.361887 15852 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:14:12.380534 15852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Generated new TSK 0
I20260924 07:14:12.381249 15852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:14:12.410805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:12.524401 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:12.524698 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:12.524761 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:12.528865 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:12.528939 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:12.529023 15864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:14:12.534453 15864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:12.535741 15864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:12.536865 15864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:12.546950 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:12.546952 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.547040 15864 server_base.cc:1034] running on GCE node
W20260924 07:14:12.546977 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.547659 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:12.548385 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:12.549568 15864 hybrid_clock.cc:648] HybridClock initialized: now 1790234052549550 us; error 33 us; skew 500 ppm
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:12.552562 15864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:14:12.553507 15864 webserver.cc:460] Webserver started at http://127.0.98.1:34833/ using document root <none> and password file <none>
I20260924 07:14:12.554087 15864 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:12.554171 15864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:12.554381 15864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:12.556571 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "884febdd7e3846e188f2f51b45d38f31"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "c0cd7520f42091323e19eda2f4c49334"
server_key_iv: "c3f60a501d5887b3cf56a759a243b606"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.557153 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "884febdd7e3846e188f2f51b45d38f31"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "c0cd7520f42091323e19eda2f4c49334"
server_key_iv: "c3f60a501d5887b3cf56a759a243b606"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.561125 15864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:14:12.563890 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.565335 15864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:14:12.565464 15864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "884febdd7e3846e188f2f51b45d38f31"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "c0cd7520f42091323e19eda2f4c49334"
server_key_iv: "c3f60a501d5887b3cf56a759a243b606"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.565582 15864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:12.582271 15864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:12.585541 15864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:12.585739 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:12.586354 15864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:12.587229 15864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:12.587278 15864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.587330 15864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:12.587358 15864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.596364 15864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35983
I20260924 07:14:12.596415 15992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35983 every 8 connection(s)
I20260924 07:14:12.597325 15864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:14:12.597780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15864
I20260924 07:14:12.597916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:14:12.598213 392 external_mini_cluster.cc:1428] Setting key eae75f0ade0abb181433c788deeeb91e
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:12.615542 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:12.599575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56527 (local address 127.0.98.62:46559)
0924 07:14:12.599731 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:12.599736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:12.600752 (+ 1016us) server_negotiation.cc:407] Connection header received
0924 07:14:12.602227 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:12.602231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:12.602300 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:12.602393 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:12.603818 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.604667 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.605984 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.606353 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.606482 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:12.611798 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:12.611802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:12.611803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:12.611836 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:12.613262 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.613870 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.613873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.613875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.613961 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.614329 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.614332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.614333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.614523 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:12.614611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:12.615189 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 07:14:12.615340 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:14:12.616441 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:12.600007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:56527)
0924 07:14:12.600561 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:12.600627 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:12.601967 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:12.602596 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:12.602604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:12.603018 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:12.603640 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:12.603659 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.604838 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.604841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:12.605774 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:12.605784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.606557 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.606561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:12.606803 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:12.607550 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:12.607589 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:12.611506 (+ 3917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:12.613422 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:12.613429 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:12.613450 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:12.613751 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:12.614057 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:12.614060 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:12.614062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:12.614227 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:12.614628 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:12.614636 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:12.614930 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 07:14:12.615115 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 07:14:12.615525 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
I20260924 07:14:12.617720 15993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:12.617973 15993 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:12.618518 15993 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:12.620275 15802 ts_manager.cc:194] Registered new tserver with Master: 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983)
I20260924 07:14:12.621415 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:14:12.626612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:12.740773 15998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:12.741082 15998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:12.741127 15998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:12.745651 15998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:12.745734 15998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:12.745800 15998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:14:12.752152 15998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:12.753328 15998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:12.754406 15998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:12.764632 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:12.764632 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.764739 15998 server_base.cc:1034] running on GCE node
W20260924 07:14:12.764894 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.765416 15998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:12.766105 15998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:12.767288 15998 hybrid_clock.cc:648] HybridClock initialized: now 1790234052767262 us; error 40 us; skew 500 ppm
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:12.770478 15998 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:14:12.771493 15998 webserver.cc:460] Webserver started at http://127.0.98.2:33213/ using document root <none> and password file <none>
I20260924 07:14:12.772054 15998 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:12.772118 15998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:12.772300 15998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:12.774380 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "64933d06e3d24a01bf32a5cb398b5925"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "f5fbc0a9b8ce0a2de6d583841c4b816a"
server_key_iv: "1dc15546b7c1120daa729e5212b78024"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.774916 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64933d06e3d24a01bf32a5cb398b5925"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "f5fbc0a9b8ce0a2de6d583841c4b816a"
server_key_iv: "1dc15546b7c1120daa729e5212b78024"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.778766 15998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 07:14:12.781308 16013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.782693 15998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:14:12.782824 15998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "64933d06e3d24a01bf32a5cb398b5925"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "f5fbc0a9b8ce0a2de6d583841c4b816a"
server_key_iv: "1dc15546b7c1120daa729e5212b78024"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.782964 15998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:12.796799 15998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:12.800170 15998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:12.800352 15998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:12.800961 15998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:12.801788 15998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:12.801846 15998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.801887 15998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:12.801913 15998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.810705 15998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39345
I20260924 07:14:12.810743 16126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39345 every 8 connection(s)
I20260924 07:14:12.811684 15998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:14:12.813217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 15998
I20260924 07:14:12.813329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:14:12.813618 392 external_mini_cluster.cc:1428] Setting key dfd1ea8392e42007ccffa9ae3661ab40
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:14:12.827129 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:12.813693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41183 (local address 127.0.98.62:46559)
0924 07:14:12.813889 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:12.813893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:12.814949 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 07:14:12.816468 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:12.816472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:12.816523 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:12.816599 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:12.817981 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.818513 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.820295 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.820559 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.820651 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:12.823553 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:12.823556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:12.823558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:12.823584 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:12.824937 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.825458 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.825460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.825462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.825537 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:12.825961 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:12.825965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:12.825967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:12.826210 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:12.826305 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:12.826827 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 07:14:12.826960 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:14:12.828187 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:12.814241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:41183)
0924 07:14:12.814795 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:12.814861 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:12.816207 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:12.816820 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:12.816827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:12.817199 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:12.817808 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:12.817827 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.818641 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.818645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:12.820147 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:12.820159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:12.820722 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:12.820726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:12.820946 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:12.821609 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:12.821633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:12.823365 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:12.825051 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:12.825057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:12.825076 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:12.825352 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:12.825617 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:12.825620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:12.825622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:12.825863 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:12.826325 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:12.826330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:12.826587 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:14:12.826747 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 07:14:12.827028 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":115,"threads_started":1}
I20260924 07:14:12.829419 16127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:12.829698 16127 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:12.830320 16127 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:12.831554 15802 ts_manager.cc:194] Registered new tserver with Master: 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
I20260924 07:14:12.832160 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:14:12.844018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:12.955955 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:12.956245 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:12.956290 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:12.960474 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:12.960543 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:12.960605 16132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:14:12.966105 16132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:12.967280 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:12.968416 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:14:12.978693 16132 server_base.cc:1034] running on GCE node
W20260924 07:14:12.978647 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:12.978647 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:12.978658 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:12.979472 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:12.980134 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:12.981323 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790234052981295 us; error 37 us; skew 500 ppm
Sep 24 07:14:12 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:12.984157 16132 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:14:12.985100 16132 webserver.cc:460] Webserver started at http://127.0.98.3:36509/ using document root <none> and password file <none>
I20260924 07:14:12.985673 16132 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:12.985734 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:12.985934 16132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:12.988124 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ed7d3409eed46d69944eb26ba536d17"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "09c1d958b83eb3941f0d84c84a5a2d38"
server_key_iv: "e89aa77ef278a49906f4f448a29eecb0"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.988715 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ed7d3409eed46d69944eb26ba536d17"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "09c1d958b83eb3941f0d84c84a5a2d38"
server_key_iv: "e89aa77ef278a49906f4f448a29eecb0"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.992558 16132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 07:14:12.995167 16147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:12.996503 16132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:14:12.996618 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "3ed7d3409eed46d69944eb26ba536d17"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "09c1d958b83eb3941f0d84c84a5a2d38"
server_key_iv: "e89aa77ef278a49906f4f448a29eecb0"
server_key_version: "encryptionkey@0"
I20260924 07:14:12.996721 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:13.031306 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:13.035035 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:13.035235 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:13.035957 16132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:13.036837 16132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:13.036890 16132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:13.036944 16132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:13.036973 16132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:13.045807 16132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32971
I20260924 07:14:13.045871 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32971 every 8 connection(s)
I20260924 07:14:13.046767 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:14:13.050756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16132
I20260924 07:14:13.050902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:14:13.051178 392 external_mini_cluster.cc:1428] Setting key 23ebf372921499be3527aee260700712
Sep 24 07:14:13 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:13.061867 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.048714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46951 (local address 127.0.98.62:46559)
0924 07:14:13.048916 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.048921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.049789 (+ 868us) server_negotiation.cc:407] Connection header received
0924 07:14:13.051475 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.051478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.051536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.051653 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:13.053226 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.053905 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.055203 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.055495 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.055580 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.058205 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:13.058208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:13.058211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:13.058240 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:13.059759 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:13.060272 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:13.060275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:13.060276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:13.060350 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:13.060659 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:13.060663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:13.060664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:13.060940 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:13.061030 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.061601 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.061719 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 07:14:13.062847 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.049102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:46951)
0924 07:14:13.049614 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.049663 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.051166 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.051957 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.051965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.052446 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:13.053077 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.053093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.054072 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.054076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.055074 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.055085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.055635 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.055638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.055847 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.056463 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:13.056482 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:13.058012 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:13.059896 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:13.059903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:13.059918 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:13.060192 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:13.060412 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:13.060415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:13.060416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:13.060559 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:13.061056 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:13.061062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:13.061355 (+ 293us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.061525 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.061873 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260924 07:14:13.064205 16261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:13.064478 16261 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:13.065059 16261 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:13.066190 15802 ts_manager.cc:194] Registered new tserver with Master: 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971)
I20260924 07:14:13.066787 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46951
I20260924 07:14:13.077589 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:14:13.086985 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.079686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.98.62:46559)
0924 07:14:13.079882 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.079886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.079991 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:14:13.080137 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.080139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.080180 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.080242 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:13.080956 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.081487 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.082328 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.082540 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.082631 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.084028 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:13.084031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:13.084033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:13.084059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:13.085286 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:13.085696 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:13.085699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:13.085701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:13.085786 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:13.086023 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:13.086026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:13.086028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:13.086289 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:13.086415 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.086633 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.086763 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:14:13.090296 15802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:14:13.093550 15802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:14:13.111070 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.103574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.0.1:45694)
0924 07:14:13.104962 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.105021 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.105158 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.106255 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.106258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.106277 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:13.106364 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.106370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.107970 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.107974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.109564 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.109574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.110620 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.110623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.110739 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.110760 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.110825 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.110894 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"mutex_wait_us":19,"thread_start_us":76,"threads_started":1}
I20260924 07:14:13.111111 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.104424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.0.98.1:35983)
0924 07:14:13.105514 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.105529 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.105549 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:14:13.105613 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.105618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.105791 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.106512 (+ 721us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:13.106674 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.107832 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.109717 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.110498 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.110721 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.110867 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.110914 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.110989 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":94,"threads_started":1}
I20260924 07:14:13.111675 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.104369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.0.1:46548)
0924 07:14:13.105565 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.105579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.105685 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.106680 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.106683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.106698 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:13.106781 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.106787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.108249 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.108253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.109438 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.109448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.110282 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.110285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.110395 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.110445 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.110561 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.111542 (+ 981us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260924 07:14:13.111966 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.104061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.0.1:38910)
0924 07:14:13.105185 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.105198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.105291 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.105953 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.105959 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.105981 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:13.106116 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.106123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.108714 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.108717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.110523 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.110531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.111618 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.111621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.111706 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.111727 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.111787 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.111851 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":93,"threads_started":1}
I20260924 07:14:13.112250 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.104548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.98.3:32971)
0924 07:14:13.106051 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.106063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.106086 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:14:13.106163 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.106168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.106366 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.106514 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:13.106915 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.108096 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.109567 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.110155 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.110252 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.110416 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.112015 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.112121 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1374,"thread_start_us":144,"threads_started":1}
I20260924 07:14:13.114094 15927 tablet_service.cc:1467] Processing CreateTablet for tablet fc40a104259c458abb708dfbec03d7a3 (DEFAULT_TABLE table=test-table [id=ef7e54efda1c4243a629c258d2451e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:13.114651 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.104833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.98.2:39345)
0924 07:14:13.105126 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.105139 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.105303 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:14:13.105406 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.105411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.105638 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.105781 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:13.107361 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.108550 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.110645 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.111514 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.111618 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.111754 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.111814 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.111901 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260924 07:14:13.114711 16195 tablet_service.cc:1467] Processing CreateTablet for tablet fc40a104259c458abb708dfbec03d7a3 (DEFAULT_TABLE table=test-table [id=ef7e54efda1c4243a629c258d2451e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:13.115183 15927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc40a104259c458abb708dfbec03d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:13.115729 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc40a104259c458abb708dfbec03d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:13.117612 16061 tablet_service.cc:1467] Processing CreateTablet for tablet fc40a104259c458abb708dfbec03d7a3 (DEFAULT_TABLE table=test-table [id=ef7e54efda1c4243a629c258d2451e63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:13.118710 16061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc40a104259c458abb708dfbec03d7a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:13.121562 16278 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap starting.
I20260924 07:14:13.123031 16279 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Bootstrap starting.
I20260924 07:14:13.124004 16278 tablet_bootstrap.cc:654] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:13.124917 16278 log.cc:824] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:13.125308 16279 tablet_bootstrap.cc:654] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:13.125784 16280 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Bootstrap starting.
I20260924 07:14:13.126580 16279 log.cc:824] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:13.128345 16280 tablet_bootstrap.cc:654] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:13.128805 16278 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: No bootstrap required, opened a new log
I20260924 07:14:13.128808 16279 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: No bootstrap required, opened a new log
I20260924 07:14:13.129016 16278 ts_tablet_manager.cc:1397] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 07:14:13.129017 16279 ts_tablet_manager.cc:1397] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:14:13.129055 16280 log.cc:824] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:13.130708 16280 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: No bootstrap required, opened a new log
I20260924 07:14:13.130891 16280 ts_tablet_manager.cc:1397] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:14:13.133144 16279 raft_consensus.cc:348] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.133459 16279 raft_consensus.cc:374] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:13.133327 16278 raft_consensus.cc:348] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.133544 16278 raft_consensus.cc:374] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:13.133551 16279 raft_consensus.cc:729] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed7d3409eed46d69944eb26ba536d17, State: Initialized, Role: FOLLOWER
I20260924 07:14:13.133639 16278 raft_consensus.cc:729] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884febdd7e3846e188f2f51b45d38f31, State: Initialized, Role: FOLLOWER
I20260924 07:14:13.134052 16278 consensus_queue.cc:260] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.134052 16279 consensus_queue.cc:260] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.134148 16280 raft_consensus.cc:348] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.134464 16280 raft_consensus.cc:374] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:13.134552 16280 raft_consensus.cc:729] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64933d06e3d24a01bf32a5cb398b5925, State: Initialized, Role: FOLLOWER
I20260924 07:14:13.134987 16261 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
I20260924 07:14:13.135043 16280 consensus_queue.cc:260] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.135118 16278 ts_tablet_manager.cc:1428] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:14:13.135118 16279 ts_tablet_manager.cc:1428] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 07:14:13.135694 15993 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
I20260924 07:14:13.135694 16127 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
I20260924 07:14:13.136039 16280 ts_tablet_manager.cc:1428] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:14:13.248718 16284 raft_consensus.cc:482] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:13.249020 16284 raft_consensus.cc:504] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.250288 16284 leader_election.cc:288] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
I20260924 07:14:13.254402 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.250809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34833 (local address 127.0.98.2:39345)
0924 07:14:13.250919 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.250924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.251238 (+ 314us) server_negotiation.cc:407] Connection header received
0924 07:14:13.251297 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.251300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.251350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.251514 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:13.251758 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.252281 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253414 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.253850 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253947 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.254010 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.254204 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.254280 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:14:13.254397 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.250715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41281 (local address 127.0.98.1:35983)
0924 07:14:13.250850 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.250855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.250874 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:14:13.251092 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.251096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.251158 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.251242 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:13.251575 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.252158 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253416 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.253826 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.253983 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.254180 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.254264 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:14:13.254398 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.250613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.98.3:41281)
0924 07:14:13.250733 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.250748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.250946 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.251283 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.251286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.251306 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:13.251466 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.251475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.252271 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.252274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.253277 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.253287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.253947 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.254056 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.254080 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.254160 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.254242 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":45}
I20260924 07:14:13.254410 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.250613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.98.3:34833)
0924 07:14:13.251096 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.251110 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:13.251229 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:13.251515 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:13.251518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:13.251533 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:13.251649 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.251654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.252393 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.252395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.253274 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:13.253284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.253974 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.253978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:13.254088 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.254106 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:14:13.254176 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:14:13.254253 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":147,"thread_start_us":124,"threads_started":1}
I20260924 07:14:13.255291 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc40a104259c458abb708dfbec03d7a3" candidate_uuid: "3ed7d3409eed46d69944eb26ba536d17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884febdd7e3846e188f2f51b45d38f31" is_pre_election: true
I20260924 07:14:13.255293 16081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc40a104259c458abb708dfbec03d7a3" candidate_uuid: "3ed7d3409eed46d69944eb26ba536d17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64933d06e3d24a01bf32a5cb398b5925" is_pre_election: true
I20260924 07:14:13.255682 16081 raft_consensus.cc:2454] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed7d3409eed46d69944eb26ba536d17 in term 0.
I20260924 07:14:13.255683 15947 raft_consensus.cc:2454] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed7d3409eed46d69944eb26ba536d17 in term 0.
I20260924 07:14:13.256201 16150 leader_election.cc:302] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925; no voters:
I20260924 07:14:13.256532 16284 raft_consensus.cc:2781] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:14:13.256639 16284 raft_consensus.cc:482] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:14:13.256682 16284 raft_consensus.cc:3037] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:13.257737 16284 raft_consensus.cc:504] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.258093 16284 leader_election.cc:288] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [CANDIDATE]: Term 1 election: Requested vote from peers 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
I20260924 07:14:13.258530 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc40a104259c458abb708dfbec03d7a3" candidate_uuid: "3ed7d3409eed46d69944eb26ba536d17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884febdd7e3846e188f2f51b45d38f31"
I20260924 07:14:13.258538 16081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc40a104259c458abb708dfbec03d7a3" candidate_uuid: "3ed7d3409eed46d69944eb26ba536d17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64933d06e3d24a01bf32a5cb398b5925"
I20260924 07:14:13.258651 16081 raft_consensus.cc:3037] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:13.258664 15947 raft_consensus.cc:3037] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:13.259781 15947 raft_consensus.cc:2454] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed7d3409eed46d69944eb26ba536d17 in term 1.
I20260924 07:14:13.259795 16081 raft_consensus.cc:2454] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed7d3409eed46d69944eb26ba536d17 in term 1.
I20260924 07:14:13.260125 16149 leader_election.cc:302] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed7d3409eed46d69944eb26ba536d17, 884febdd7e3846e188f2f51b45d38f31; no voters:
I20260924 07:14:13.260317 16284 raft_consensus.cc:2781] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:14:13.260542 16284 raft_consensus.cc:686] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed7d3409eed46d69944eb26ba536d17, State: Running, Role: LEADER
I20260924 07:14:13.260875 16284 consensus_queue.cc:237] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:13.264496 15801 catalog_manager.cc:5495] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ed7d3409eed46d69944eb26ba536d17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } health_report { overall_health: HEALTHY } } }
W20260924 07:14:13.297724 16262 tablet.cc:2369] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:14:13.301224 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.297143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.98.3:32971)
0924 07:14:13.297305 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:13.297309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:13.297325 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:14:13.297399 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:13.297401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:13.297457 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:13.297591 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:13.297832 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.298460 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.299421 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:13.299781 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:13.299889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:13.300322 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:13.300842 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:13.300939 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:13.301030 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 07:14:13.301090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:14:13.307461 16081 raft_consensus.cc:1261] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Refusing update from remote peer 3ed7d3409eed46d69944eb26ba536d17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:13.307461 15947 raft_consensus.cc:1261] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Refusing update from remote peer 3ed7d3409eed46d69944eb26ba536d17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:13.308197 16284 consensus_queue.cc:1035] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:14:13.308432 16288 consensus_queue.cc:1035] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:14:13.312845 16128 tablet.cc:2369] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:14:13.318131 16291 mvcc.cc:205] Tried to move back new op lower bound from 7332798682339270656 to 7332798682159091712. Current Snapshot: MvccSnapshot[applied={T|T < 7332798682339270656}]
I20260924 07:14:13.326915 15801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:14:13.327114 15801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:14:13.329654 15801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Sending DeleteTablet for 3 replicas of tablet fc40a104259c458abb708dfbec03d7a3
I20260924 07:14:13.330503 16061 tablet_service.cc:1514] Processing DeleteTablet for tablet fc40a104259c458abb708dfbec03d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38910
I20260924 07:14:13.330566 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet fc40a104259c458abb708dfbec03d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46548
I20260924 07:14:13.330981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15864
I20260924 07:14:13.330998 15927 tablet_service.cc:1514] Processing DeleteTablet for tablet fc40a104259c458abb708dfbec03d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45694
I20260924 07:14:13.331756 16301 tablet_replica.cc:330] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: stopping tablet replica
I20260924 07:14:13.332090 16301 raft_consensus.cc:2229] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:14:13.332435 16301 raft_consensus.cc:2258] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:14:13.332773 16300 tablet_replica.cc:330] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: stopping tablet replica
I20260924 07:14:13.333043 16300 raft_consensus.cc:2229] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:14:13.333338 16300 raft_consensus.cc:2258] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:14:13.336782 16300 ts_tablet_manager.cc:1905] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:14:13.337105 16301 ts_tablet_manager.cc:1905] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:14:13.337577 15787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35983 (error 108)
I20260924 07:14:13.338464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15998
I20260924 07:14:13.339700 16301 ts_tablet_manager.cc:1918] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:14:13.339789 16301 log.cc:1196] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/fc40a104259c458abb708dfbec03d7a3
I20260924 07:14:13.340051 16301 ts_tablet_manager.cc:1939] T fc40a104259c458abb708dfbec03d7a3 P 3ed7d3409eed46d69944eb26ba536d17: Deleting consensus metadata
I20260924 07:14:13.341496 15788 catalog_manager.cc:4841] TS 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): tablet fc40a104259c458abb708dfbec03d7a3 (table test-table [id=ef7e54efda1c4243a629c258d2451e63]) successfully deleted
I20260924 07:14:13.342021 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.341671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.0.1:45704)
0924 07:14:13.341857 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.341931 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260924 07:14:13.342206 15787 catalog_manager.cc:4584] TS 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc40a104259c458abb708dfbec03d7a3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35983: connect: Connection refused (error 111)
I20260924 07:14:13.345077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16132
I20260924 07:14:13.345286 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:13.345038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.0.1:38918)
0924 07:14:13.345160 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:13.345209 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:14:13.350471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 15770
I20260924 07:14:33.356484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--webserver_interface=127.0.98.62
--webserver_port=44979
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:33.473210 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:33.473526 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:33.473572 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:33.477783 16303 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:14:33.477864 16303 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:14:33.477886 16303 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:14:33.477905 16303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:33.477929 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:33.477952 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:33.477972 16303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:33.477994 16303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:33.483732 16303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:33.484891 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:33.485998 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:33.500787 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:33.500837 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:33.501109 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:33.501312 16303 server_base.cc:1034] running on GCE node
I20260924 07:14:33.502128 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:33.503767 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:33.504956 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790234073504931 us; error 43 us; skew 500 ppm
Sep 24 07:14:33 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:33.508564 16303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:14:33.509847 16303 webserver.cc:460] Webserver started at http://127.0.98.62:44979/ using document root <none> and password file <none>
I20260924 07:14:33.510591 16303 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:33.510694 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:33.516160 16303 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:33.519454 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:33.521292 16303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:14:33.521468 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "8099982d493d43a3aea0638102d4343f"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "97960cce7cdca35541746cf5768b7b7f"
server_key_iv: "bc4f86640c1b09be32c52a3587f759d8"
server_key_version: "encryptionkey@0"
I20260924 07:14:33.522342 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:33.553216 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:33.557494 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:33.557751 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:33.566912 16303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46559
I20260924 07:14:33.567234 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46559 every 8 connection(s)
I20260924 07:14:33.568414 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:14:33.573911 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:14:33.575686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16303
I20260924 07:14:33.576104 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap starting.
I20260924 07:14:33.576231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35983
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=34833
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:33.594308 16372 log.cc:824] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:33.603458 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:14:33.603840 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap complete.
I20260924 07:14:33.607954 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:33.608414 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Initialized, Role: FOLLOWER
I20260924 07:14:33.608958 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:33.609099 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:14:33.609177 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:14:33.609282 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:14:33.611590 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:33.611956 16372 leader_election.cc:302] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8099982d493d43a3aea0638102d4343f; no voters:
I20260924 07:14:33.612272 16372 leader_election.cc:288] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:14:33.612375 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:14:33.612627 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 LEADER]: Becoming Leader. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Running, Role: LEADER
I20260924 07:14:33.612967 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:33.613675 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:33.614411 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:33.614567 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:33.614873 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8099982d493d43a3aea0638102d4343f. Latest consensus state: current_term: 2 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:33.614974 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:33.615876 16384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:14:33.620849 16384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:14:33.621532 16384 catalog_manager.cc:1259] Loaded cluster ID: 8e09f01f8fdc4491ad2b0f284555270d
I20260924 07:14:33.621598 16384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:14:33.624305 16384 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:14:33.625077 16384 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Loaded TSK: 0
I20260924 07:14:33.645273 16384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Generated new TSK 1
I20260924 07:14:33.645622 16384 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:14:33.710135 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:33.710508 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:33.710579 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:33.715324 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:33.715447 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:33.715543 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:14:33.721997 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34833
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:33.723529 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:33.724858 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:33.737080 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:33.737032 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:33.737047 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:33.737386 16374 server_base.cc:1034] running on GCE node
I20260924 07:14:33.738124 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:33.739172 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:33.740391 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790234073740361 us; error 76 us; skew 500 ppm
Sep 24 07:14:33 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:33.744180 16374 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:14:33.745415 16374 webserver.cc:460] Webserver started at http://127.0.98.1:34833/ using document root <none> and password file <none>
I20260924 07:14:33.746109 16374 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:33.746202 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:33.751472 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:33.754490 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:33.756322 16374 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:33.756475 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "884febdd7e3846e188f2f51b45d38f31"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "c0cd7520f42091323e19eda2f4c49334"
server_key_iv: "c3f60a501d5887b3cf56a759a243b606"
server_key_version: "encryptionkey@0"
I20260924 07:14:33.757242 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:33.775621 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:33.779446 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:33.779649 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:33.780375 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:33.781663 16416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:14:33.786772 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:14:33.786877 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260924 07:14:33.786960 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:14:33.789336 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:14:33.789395 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 07:14:33.789824 16416 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap starting.
I20260924 07:14:33.801095 16374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35983
I20260924 07:14:33.801216 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35983 every 8 connection(s)
I20260924 07:14:33.802321 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:14:33.805040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16374
I20260924 07:14:33.805660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39345
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=33213
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:33.809784 16416 log.cc:824] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:14:33 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:33.818660 16416 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:14:33.819116 16416 tablet_bootstrap.cc:492] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap complete.
I20260924 07:14:33.820462 16416 ts_tablet_manager.cc:1397] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.012s
I20260924 07:14:33.823585 16416 raft_consensus.cc:348] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:33.824046 16416 raft_consensus.cc:729] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884febdd7e3846e188f2f51b45d38f31, State: Initialized, Role: FOLLOWER
I20260924 07:14:33.824590 16416 consensus_queue.cc:260] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } }
I20260924 07:14:33.825676 16416 ts_tablet_manager.cc:1428] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:14:33.826073 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:33.804674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:47603)
0924 07:14:33.805880 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:33.805961 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:33.806930 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:33.808504 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:33.808516 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:33.809101 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:33.809793 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:33.809811 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.811822 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.811827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:33.813292 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:33.813309 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.814565 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.814570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:33.814857 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:33.815774 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:33.815796 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:33.818141 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:33.821332 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:33.821341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:33.821365 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:33.821867 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:33.822367 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:33.822371 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:33.822373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:33.822580 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:33.823412 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:33.823421 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:33.823853 (+ 432us) client_negotiation.cc:769] Sending connection context
0924 07:14:33.824091 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 07:14:33.824614 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":169,"threads_started":1}
I20260924 07:14:33.827428 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:33.804852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47603 (local address 127.0.98.62:46559)
0924 07:14:33.805449 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:33.805469 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:33.806103 (+ 634us) server_negotiation.cc:407] Connection header received
0924 07:14:33.807475 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:33.807492 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:33.807974 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:33.809102 (+ 1128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:33.810030 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.811619 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.813504 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.814370 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.814564 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:33.818499 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:33.818509 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:33.818530 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:33.818576 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:33.821127 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:33.822048 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:33.822055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:33.822064 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:33.822248 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:33.822692 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:33.822696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:33.822698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:33.823249 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:33.823439 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:33.826122 (+ 2683us) server_negotiation.cc:299] Negotiation successful
0924 07:14:33.826519 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":124,"threads_started":1}
I20260924 07:14:33.830260 16524 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:33.830608 16524 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:33.831353 16524 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:33.834859 16336 ts_manager.cc:194] Registered new tserver with Master: 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983)
I20260924 07:14:33.838861 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47603
I20260924 07:14:33.842180 16524 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
Sep 24 07:14:33 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:14:33.851037 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:33.838491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.0.98.1:35983)
0924 07:14:33.839028 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:33.839042 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:33.839690 (+ 648us) server_negotiation.cc:407] Connection header received
0924 07:14:33.839954 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:33.839960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:33.840188 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:33.840338 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:33.841270 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.842339 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.843701 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.844205 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.844297 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:33.847223 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:33.847233 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:33.847246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:33.847269 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:33.849072 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:33.849591 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:33.849596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:33.849597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:33.849673 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:33.849947 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:33.849950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:33.849952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:33.850206 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:33.850327 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:33.850688 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 07:14:33.850857 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":404,"thread_start_us":138,"threads_started":1}
I20260924 07:14:33.851037 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:33.838270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.0.1:39246)
0924 07:14:33.839451 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:33.839523 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:33.839744 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:33.840392 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:33.840399 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:33.840926 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:33.841108 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:33.841118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.842482 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.842486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:33.843563 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:33.843575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:33.844313 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:33.844340 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:33.844468 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:33.845127 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:33.845142 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:33.847067 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:33.849203 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:33.849215 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:33.849223 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:33.849478 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:33.849764 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:33.849767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:33.849769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:33.849846 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:33.850316 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:33.850322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:33.850456 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:14:33.850585 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 07:14:33.850842 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":135,"threads_started":1}
I20260924 07:14:33.852712 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet fc40a104259c458abb708dfbec03d7a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 07:14:33.853726 16533 tablet_replica.cc:330] stopping tablet replica
I20260924 07:14:33.854156 16533 raft_consensus.cc:2229] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:14:33.854488 16533 raft_consensus.cc:2258] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:14:33.855930 16533 ts_tablet_manager.cc:1905] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:14:33.862183 16533 ts_tablet_manager.cc:1918] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:14:33.862365 16533 log.cc:1196] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal/wals/fc40a104259c458abb708dfbec03d7a3
I20260924 07:14:33.862654 16533 ts_tablet_manager.cc:1939] T fc40a104259c458abb708dfbec03d7a3 P 884febdd7e3846e188f2f51b45d38f31: Deleting consensus metadata
I20260924 07:14:33.865526 16321 catalog_manager.cc:4841] TS 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983): tablet fc40a104259c458abb708dfbec03d7a3 (table test-table [id=ef7e54efda1c4243a629c258d2451e63]) successfully deleted
W20260924 07:14:33.946280 16528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:33.946575 16528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:33.946619 16528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:33.950781 16528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:33.950857 16528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:33.950932 16528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:14:33.956503 16528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33213
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:33.957661 16528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:33.958743 16528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:33.969193 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:33.969242 16528 server_base.cc:1034] running on GCE node
W20260924 07:14:33.969223 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:33.969193 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:33.969921 16528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:33.970561 16528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:33.971741 16528 hybrid_clock.cc:648] HybridClock initialized: now 1790234073971722 us; error 30 us; skew 500 ppm
Sep 24 07:14:33 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:33.979351 16528 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:14:33.980440 16528 webserver.cc:460] Webserver started at http://127.0.98.2:33213/ using document root <none> and password file <none>
I20260924 07:14:33.981070 16528 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:33.981156 16528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:33.985450 16528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:14:33.988132 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:33.989565 16528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:14:33.989677 16528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "64933d06e3d24a01bf32a5cb398b5925"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "f5fbc0a9b8ce0a2de6d583841c4b816a"
server_key_iv: "1dc15546b7c1120daa729e5212b78024"
server_key_version: "encryptionkey@0"
I20260924 07:14:33.990259 16528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:34.000859 16528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:34.003918 16528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:34.004096 16528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:34.004750 16528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:34.005832 16555 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 07:14:34.009939 16555 ts_tablet_manager.cc:1852] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 07:14:34.010375 16555 ts_tablet_manager.cc:1905] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:14:34.014896 16555 ts_tablet_manager.cc:1918] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:14:34.015051 16555 log.cc:1196] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/fc40a104259c458abb708dfbec03d7a3
I20260924 07:14:34.015419 16555 ts_tablet_manager.cc:1939] T fc40a104259c458abb708dfbec03d7a3 P 64933d06e3d24a01bf32a5cb398b5925: Deleting consensus metadata
I20260924 07:14:34.015764 16528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 07:14:34.015829 16528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260924 07:14:34.015894 16528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:34.015925 16528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:34.024893 16528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39345
I20260924 07:14:34.024945 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39345 every 8 connection(s)
I20260924 07:14:34.025882 16528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:14:34 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:34.035624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16528
I20260924 07:14:34.036029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=36509
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:34.039407 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.027841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49013 (local address 127.0.98.62:46559)
0924 07:14:34.027970 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.027974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.028851 (+ 877us) server_negotiation.cc:407] Connection header received
0924 07:14:34.029810 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.029814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.029874 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.029946 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:34.031168 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.031858 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.033051 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.033277 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.033367 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.035874 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:34.035877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:34.035879 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:34.035906 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:34.037373 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.037873 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.037877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.037879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.037950 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.038263 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.038266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.038268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.038455 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:34.038538 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.039069 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.039209 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:14:34.040302 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.028218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:49013)
0924 07:14:34.028705 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.028747 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.029604 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.030133 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.030140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.030504 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:34.031027 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.031042 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.032006 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.032010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.032925 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.032935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.033421 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.033425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.033598 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.034186 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:34.034203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:34.035701 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:34.037491 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:34.037497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:34.037524 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:34.037778 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:34.038032 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:34.038035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:34.038037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:34.038178 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:34.038559 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:34.038566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:34.038826 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.039007 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.039356 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260924 07:14:34.041671 16663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:34.041922 16663 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:34.042448 16663 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:34.043643 16336 ts_manager.cc:194] Registered new tserver with Master: 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
I20260924 07:14:34.044420 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49013
W20260924 07:14:34.152454 16666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:34.152734 16666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:34.152777 16666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:34.156867 16666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:34.156939 16666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:34.157006 16666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:14:34.162670 16666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36509
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:34.163811 16666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:34.164847 16666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:34.174476 16671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:34.174643 16666 server_base.cc:1034] running on GCE node
W20260924 07:14:34.174436 16674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:34.174453 16672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:34.175275 16666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:34.176008 16666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:34.177186 16666 hybrid_clock.cc:648] HybridClock initialized: now 1790234074177158 us; error 41 us; skew 500 ppm
Sep 24 07:14:34 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:34.180084 16666 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:14:34.181093 16666 webserver.cc:460] Webserver started at http://127.0.98.3:36509/ using document root <none> and password file <none>
I20260924 07:14:34.181649 16666 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:34.181733 16666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:34.185766 16666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:14:34.188006 16681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:34.189327 16666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:14:34.189460 16666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "3ed7d3409eed46d69944eb26ba536d17"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "09c1d958b83eb3941f0d84c84a5a2d38"
server_key_iv: "e89aa77ef278a49906f4f448a29eecb0"
server_key_version: "encryptionkey@0"
I20260924 07:14:34.190086 16666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:34.211488 16666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:34.215173 16666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:34.215366 16666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:34.215981 16666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:34.216804 16666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:34.216848 16666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:34.216889 16666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:34.216918 16666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:34.225493 16666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32971
I20260924 07:14:34.225528 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32971 every 8 connection(s)
I20260924 07:14:34.226501 16666 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:14:34.232841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16666
Sep 24 07:14:34 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:34.240221 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.228322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59693 (local address 127.0.98.62:46559)
0924 07:14:34.228466 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.228469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.229320 (+ 851us) server_negotiation.cc:407] Connection header received
0924 07:14:34.230585 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.230588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.230645 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.230719 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:34.231934 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.232448 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.233628 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.233919 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.234015 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.236886 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:34.236890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:34.236893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:34.236918 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:34.238225 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.238715 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.238719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.238721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.238790 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.239114 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.239118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.239120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.239323 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:34.239457 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.239996 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.240096 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:14:34.241161 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.228728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:59693)
0924 07:14:34.229166 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.229206 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.230387 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.230907 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.230913 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.231269 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:34.231802 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.231815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.232562 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.232564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.233516 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.233525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.234084 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.234088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.234300 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.234885 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:34.234905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:34.236738 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:34.238339 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:34.238345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:34.238362 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:34.238620 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:34.238877 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:34.238880 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:34.238882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:34.239026 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:34.239500 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:34.239507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:34.239757 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.239943 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.240318 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260924 07:14:34.241859 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.233491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48188 (local address 127.0.98.62:46559)
0924 07:14:34.233751 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.233755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.233928 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:14:34.234067 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.234071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.234120 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.234221 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:34.234883 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.235644 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.236792 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.237096 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.237201 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.238406 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:34.238409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:34.238411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:34.238504 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:34.239833 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.240382 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.240386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.240388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.240467 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.240723 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.240726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.240728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.240878 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:34.241073 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.241231 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.241719 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260924 07:14:34.242398 16795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46559
I20260924 07:14:34.242647 16795 heartbeater.cc:464] Registering TS with master...
I20260924 07:14:34.243151 16795 heartbeater.cc:511] Master 127.0.98.62:46559 requested a full tablet report, sending...
I20260924 07:14:34.244230 16336 ts_manager.cc:194] Registered new tserver with Master: 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971)
I20260924 07:14:34.244747 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59693
I20260924 07:14:34.255573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:14:34.259647 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.256221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.98.62:46559)
0924 07:14:34.256375 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.256380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.256539 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:14:34.256711 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.256714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.256819 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.256911 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:34.257144 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.257673 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.258504 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.258731 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.258822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.259099 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:34.259242 (+ 143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 07:14:34.259511 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48196: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 07:14:34.259716 16798 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48196: token expired
W20260924 07:14:34.260010 16266 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:46559)
I20260924 07:14:34.267128 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.260375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.98.62:46559)
0924 07:14:34.260468 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.260471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.260483 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:14:34.260596 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.260599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.260635 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.260690 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:34.261289 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.261821 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.262625 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.262836 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.262919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.264288 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:34.264292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:34.264295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:34.264320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:34.265631 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.266011 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.266015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.266017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.266090 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:34.266331 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:34.266334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:34.266336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:34.266518 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:34.266630 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.266851 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.266969 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:14:34.269693 16336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:14:34.273048 16336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:14:34.280884 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 30db659c5ba84b3a9b059119da882203 (DEFAULT_TABLE table=test-table [id=8bacb2c9635d4b97b2c15ed9b47529ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:34.281498 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30db659c5ba84b3a9b059119da882203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:34.284663 16416 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap starting.
I20260924 07:14:34.285048 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.279897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.0.1:45818)
0924 07:14:34.280035 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.280050 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.280122 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.281449 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.281452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.281470 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:34.281653 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.281660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.282741 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.282744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.283757 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.283766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.284629 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.284631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.284722 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.284771 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.284832 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.284917 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 07:14:34.285641 16416 tablet_bootstrap.cc:654] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:34.285787 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.280414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.0.98.3:32971)
0924 07:14:34.280962 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.280973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.280990 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:14:34.281069 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.281074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.281296 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.281457 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:34.281756 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.282613 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.283868 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.284490 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.284631 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.284921 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.284976 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.285084 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":159,"threads_started":1}
I20260924 07:14:34.286139 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.280131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.0.1:39842)
0924 07:14:34.280777 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.280791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.280888 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.281740 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.281743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.281767 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:34.281851 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.281857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.283347 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.283351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.284458 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.284468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.285748 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.285751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.285841 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.285881 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.285945 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.286015 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":121,"threads_started":1}
I20260924 07:14:34.286604 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.280382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.0.98.2:39345)
0924 07:14:34.280706 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.280719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.280981 (+ 262us) server_negotiation.cc:407] Connection header received
0924 07:14:34.281069 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.281074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.281296 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.281457 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:34.281975 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.283218 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.284601 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.285402 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.285527 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.285682 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.286303 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.286405 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":99,"threads_started":1}
I20260924 07:14:34.287115 16416 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: No bootstrap required, opened a new log
I20260924 07:14:34.287181 16416 ts_tablet_manager.cc:1397] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 07:14:34.287652 16416 raft_consensus.cc:348] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.287781 16416 raft_consensus.cc:374] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:34.287827 16416 raft_consensus.cc:729] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884febdd7e3846e188f2f51b45d38f31, State: Initialized, Role: FOLLOWER
I20260924 07:14:34.287992 16416 consensus_queue.cc:260] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.287957 16597 tablet_service.cc:1467] Processing CreateTablet for tablet 30db659c5ba84b3a9b059119da882203 (DEFAULT_TABLE table=test-table [id=8bacb2c9635d4b97b2c15ed9b47529ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:34.289166 16597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30db659c5ba84b3a9b059119da882203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:34.289125 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 30db659c5ba84b3a9b059119da882203 (DEFAULT_TABLE table=test-table [id=8bacb2c9635d4b97b2c15ed9b47529ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:14:34.290093 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30db659c5ba84b3a9b059119da882203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:34.290446 16416 ts_tablet_manager.cc:1428] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:14:34.294142 16555 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Bootstrap starting.
I20260924 07:14:34.296527 16555 tablet_bootstrap.cc:654] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:34.297389 16808 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Bootstrap starting.
I20260924 07:14:34.297469 16555 log.cc:824] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:34.299290 16555 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: No bootstrap required, opened a new log
I20260924 07:14:34.299500 16555 ts_tablet_manager.cc:1397] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:14:34.300012 16808 tablet_bootstrap.cc:654] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:34.300837 16808 log.cc:824] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:34.302685 16808 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: No bootstrap required, opened a new log
I20260924 07:14:34.302863 16808 ts_tablet_manager.cc:1397] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:14:34.302804 16555 raft_consensus.cc:348] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.303046 16555 raft_consensus.cc:374] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:34.303102 16555 raft_consensus.cc:729] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64933d06e3d24a01bf32a5cb398b5925, State: Initialized, Role: FOLLOWER
I20260924 07:14:34.303556 16555 consensus_queue.cc:260] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
W20260924 07:14:34.303737 16525 tablet.cc:2369] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:14:34.304394 16555 ts_tablet_manager.cc:1428] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:14:34.304771 16663 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
I20260924 07:14:34.306480 16808 raft_consensus.cc:348] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.306752 16808 raft_consensus.cc:374] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:14:34.306820 16808 raft_consensus.cc:729] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed7d3409eed46d69944eb26ba536d17, State: Initialized, Role: FOLLOWER
I20260924 07:14:34.307300 16808 consensus_queue.cc:260] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.308372 16795 heartbeater.cc:503] Master 127.0.98.62:46559 was elected leader, sending a full tablet report...
I20260924 07:14:34.308372 16808 ts_tablet_manager.cc:1428] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:14:34.373920 16811 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:34.374168 16811 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.375280 16811 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983)
I20260924 07:14:34.379788 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.375714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44367 (local address 127.0.98.3:32971)
0924 07:14:34.375833 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.375837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.376063 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:14:34.376137 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.376140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.376186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.376268 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:34.376720 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.377519 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.378750 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.379163 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.379258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.379319 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.379595 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.379663 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:14:34.379792 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.375618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.98.2:46491)
0924 07:14:34.375726 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.375741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.375835 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.376283 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.376286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.376326 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:34.376620 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.376627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.377701 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.377705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.378561 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.378571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.379285 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.379289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.379458 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.379493 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.379558 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.379642 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":219,"spinlock_wait_cycles":32384}
I20260924 07:14:34.379806 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.375619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.98.2:44367)
0924 07:14:34.375979 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.375988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:34.376065 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:34.376283 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:34.376286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:34.376324 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:14:34.376599 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.376608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.377633 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.377638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.378568 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:34.378579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.379292 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.379299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:34.379478 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.379499 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:14:34.379566 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:14:34.379642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":236,"spinlock_wait_cycles":9088,"thread_start_us":84,"threads_started":1}
I20260924 07:14:34.379796 16813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.375716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46491 (local address 127.0.98.1:35983)
0924 07:14:34.376008 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.376012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.376025 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:14:34.376093 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.376096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.376163 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.376269 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:14:34.376720 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.377519 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.378680 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.379158 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.379255 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.379441 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.379587 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.379660 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":116,"threads_started":1}
I20260924 07:14:34.380606 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30db659c5ba84b3a9b059119da882203" candidate_uuid: "64933d06e3d24a01bf32a5cb398b5925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed7d3409eed46d69944eb26ba536d17" is_pre_election: true
I20260924 07:14:34.380724 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30db659c5ba84b3a9b059119da882203" candidate_uuid: "64933d06e3d24a01bf32a5cb398b5925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884febdd7e3846e188f2f51b45d38f31" is_pre_election: true
I20260924 07:14:34.380936 16749 raft_consensus.cc:2454] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64933d06e3d24a01bf32a5cb398b5925 in term 0.
I20260924 07:14:34.381006 16478 raft_consensus.cc:2454] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64933d06e3d24a01bf32a5cb398b5925 in term 0.
I20260924 07:14:34.381408 16551 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925, 884febdd7e3846e188f2f51b45d38f31; no voters:
I20260924 07:14:34.381690 16811 raft_consensus.cc:2781] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:14:34.381775 16811 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:14:34.381820 16811 raft_consensus.cc:3037] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:34.382789 16811 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.383121 16811 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [CANDIDATE]: Term 1 election: Requested vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983)
I20260924 07:14:34.383464 16749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30db659c5ba84b3a9b059119da882203" candidate_uuid: "64933d06e3d24a01bf32a5cb398b5925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed7d3409eed46d69944eb26ba536d17"
I20260924 07:14:34.383489 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30db659c5ba84b3a9b059119da882203" candidate_uuid: "64933d06e3d24a01bf32a5cb398b5925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "884febdd7e3846e188f2f51b45d38f31"
I20260924 07:14:34.383595 16478 raft_consensus.cc:3037] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:34.383657 16749 raft_consensus.cc:3037] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:34.384550 16478 raft_consensus.cc:2454] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64933d06e3d24a01bf32a5cb398b5925 in term 1.
I20260924 07:14:34.384686 16749 raft_consensus.cc:2454] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64933d06e3d24a01bf32a5cb398b5925 in term 1.
I20260924 07:14:34.384868 16550 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64933d06e3d24a01bf32a5cb398b5925, 884febdd7e3846e188f2f51b45d38f31; no voters:
I20260924 07:14:34.385032 16811 raft_consensus.cc:2781] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:14:34.385262 16811 raft_consensus.cc:686] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 LEADER]: Becoming Leader. State: Replica: 64933d06e3d24a01bf32a5cb398b5925, State: Running, Role: LEADER
I20260924 07:14:34.385628 16811 consensus_queue.cc:237] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:34.388680 16335 catalog_manager.cc:5495] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "64933d06e3d24a01bf32a5cb398b5925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } health_report { overall_health: UNKNOWN } } }
I20260924 07:14:34.413211 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.409633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.0.98.2:39345)
0924 07:14:34.409760 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:34.409766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:34.409779 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:14:34.409854 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:34.409856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:34.409910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:34.410002 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:34.410292 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.410802 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.411723 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:34.411939 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:34.412026 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:34.412361 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:34.412812 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:34.412929 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:34.412993 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 07:14:34.413062 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:14:34.419734 16749 raft_consensus.cc:1261] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Refusing update from remote peer 64933d06e3d24a01bf32a5cb398b5925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:34.419731 16478 raft_consensus.cc:1261] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Refusing update from remote peer 64933d06e3d24a01bf32a5cb398b5925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:34.420439 16815 consensus_queue.cc:1035] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:14:34.420624 16811 consensus_queue.cc:1035] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:14:34.426913 16818 mvcc.cc:205] Tried to move back new op lower bound from 7332798768815435776 to 7332798768686034944. Current Snapshot: MvccSnapshot[applied={T|T < 7332798768815435776}]
I20260924 07:14:34.430238 16820 mvcc.cc:205] Tried to move back new op lower bound from 7332798768815435776 to 7332798768686034944. Current Snapshot: MvccSnapshot[applied={T|T < 7332798768815435776 or (T in {7332798768815435776})}]
I20260924 07:14:34.435175 16335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:14:34.435333 16335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:14:34.437652 16335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Sending DeleteTablet for 3 replicas of tablet 30db659c5ba84b3a9b059119da882203
I20260924 07:14:34.438345 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 30db659c5ba84b3a9b059119da882203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39246
I20260924 07:14:34.438426 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 30db659c5ba84b3a9b059119da882203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45818
I20260924 07:14:34.438783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16374
I20260924 07:14:34.439109 16597 tablet_service.cc:1514] Processing DeleteTablet for tablet 30db659c5ba84b3a9b059119da882203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:14:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39842
I20260924 07:14:34.440433 16829 tablet_replica.cc:330] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: stopping tablet replica
I20260924 07:14:34.440812 16829 raft_consensus.cc:2229] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:14:34.441165 16829 raft_consensus.cc:2258] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:14:34.441569 16830 tablet_replica.cc:330] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: stopping tablet replica
I20260924 07:14:34.441855 16830 raft_consensus.cc:2229] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:14:34.442224 16830 raft_consensus.cc:2258] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:14:34.443044 16829 ts_tablet_manager.cc:1905] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:14:34.444033 16830 ts_tablet_manager.cc:1905] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:14:34.445943 16829 ts_tablet_manager.cc:1918] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:14:34.446120 16829 log.cc:1196] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/30db659c5ba84b3a9b059119da882203
W20260924 07:14:34.446203 16321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35983 (error 108)
I20260924 07:14:34.446630 16829 ts_tablet_manager.cc:1939] T 30db659c5ba84b3a9b059119da882203 P 3ed7d3409eed46d69944eb26ba536d17: Deleting consensus metadata
I20260924 07:14:34.446732 16830 ts_tablet_manager.cc:1918] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:14:34.446812 16830 log.cc:1196] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/30db659c5ba84b3a9b059119da882203
I20260924 07:14:34.447058 16830 ts_tablet_manager.cc:1939] T 30db659c5ba84b3a9b059119da882203 P 64933d06e3d24a01bf32a5cb398b5925: Deleting consensus metadata
I20260924 07:14:34.447557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16528
I20260924 07:14:34.449113 16322 catalog_manager.cc:4841] TS 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): tablet 30db659c5ba84b3a9b059119da882203 (table test-table [id=8bacb2c9635d4b97b2c15ed9b47529ba]) successfully deleted
I20260924 07:14:34.449508 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.449007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35983 (local address 127.0.0.1:39250)
0924 07:14:34.449359 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.449426 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
W20260924 07:14:34.449661 16321 catalog_manager.cc:4584] TS 884febdd7e3846e188f2f51b45d38f31 (127.0.98.1:35983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30db659c5ba84b3a9b059119da882203: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35983: connect: Connection refused (error 111)
I20260924 07:14:34.453011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16666
I20260924 07:14:34.454030 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:34.453739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.0.1:39858)
0924 07:14:34.453899 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:34.453946 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 07:14:34.459784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16303
I20260924 07:14:54.465870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--webserver_interface=127.0.98.62
--webserver_port=44979
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:54.578889 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:54.579200 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:54.579260 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:54.583616 16832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:14:54.583684 16832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:14:54.583702 16832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:14:54.583715 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:54.583734 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:54.583750 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:54.583766 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:54.583792 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:54.589275 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:54.590360 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:54.591342 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:54.600943 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:54.601037 16832 server_base.cc:1034] running on GCE node
W20260924 07:14:54.600982 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:54.600982 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:54.602023 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:54.603243 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:54.604430 16832 hybrid_clock.cc:648] HybridClock initialized: now 1790234094604399 us; error 64 us; skew 500 ppm
Sep 24 07:14:54 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:54.612697 16832 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:14:54.614058 16832 webserver.cc:460] Webserver started at http://127.0.98.62:44979/ using document root <none> and password file <none>
I20260924 07:14:54.614776 16832 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:54.614849 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:54.620167 16832 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:54.623186 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:54.624888 16832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:14:54.625053 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "8099982d493d43a3aea0638102d4343f"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "97960cce7cdca35541746cf5768b7b7f"
server_key_iv: "bc4f86640c1b09be32c52a3587f759d8"
server_key_version: "encryptionkey@0"
I20260924 07:14:54.625829 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:54.647013 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:54.651186 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:54.651463 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:54.660069 16832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46559
I20260924 07:14:54.660455 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46559 every 8 connection(s)
I20260924 07:14:54.661432 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:14:54.663661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16832
I20260924 07:14:54.664193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35983
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=34833
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:54.667227 16900 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:14:54.669153 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap starting.
I20260924 07:14:54.687115 16900 log.cc:824] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:54.698920 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:14:54.699410 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Bootstrap complete.
I20260924 07:14:54.703629 16900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:54.704092 16900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Initialized, Role: FOLLOWER
I20260924 07:14:54.704640 16900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:54.704770 16900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:14:54.704840 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:14:54.704941 16900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:14:54.707144 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:54.707545 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8099982d493d43a3aea0638102d4343f; no voters:
I20260924 07:14:54.707885 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:14:54.707993 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:14:54.708230 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [term 3 LEADER]: Becoming Leader. State: Replica: 8099982d493d43a3aea0638102d4343f, State: Running, Role: LEADER
I20260924 07:14:54.708688 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } }
I20260924 07:14:54.709399 16900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:54.710418 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:54.710649 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:54.711020 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8099982d493d43a3aea0638102d4343f. Latest consensus state: current_term: 3 leader_uuid: "8099982d493d43a3aea0638102d4343f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8099982d493d43a3aea0638102d4343f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46559 } } }
I20260924 07:14:54.711118 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:54.711190 16913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:14:54.717059 16913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:14:54.717825 16913 catalog_manager.cc:1259] Loaded cluster ID: 8e09f01f8fdc4491ad2b0f284555270d
I20260924 07:14:54.717905 16913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:14:54.720767 16913 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:14:54.721638 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Loaded TSK: 0
I20260924 07:14:54.721740 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8099982d493d43a3aea0638102d4343f: Loaded TSK: 1
F20260924 07:14:54.722632 16913 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f83105c29ad google::LogMessage::Fail() at ??:0
@ 0x7f83105c588a google::LogMessage::SendToLog() at ??:0
@ 0x7f83105c24de google::LogMessage::Flush() at ??:0
@ 0x7f83105c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f831328618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f831327ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f831af277bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f831af0199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f831af02178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f831af43df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f831af6282b std::function<>::operator()() at ??:0
@ 0x7f831af01b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f831af02f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f831aeff579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f831af43640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f83128143d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8312814c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f83128163f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8312805512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8313e496db start_thread at ??:0
@ 0x7f830f0c061f clone at ??:0
W20260924 07:14:54.834429 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:54.834800 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:54.834848 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:54.839403 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:54.839515 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:54.839596 16902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:14:54.845995 16902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34833
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:54.847600 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:54.848984 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:54.861465 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:54.861639 16902 server_base.cc:1034] running on GCE node
W20260924 07:14:54.861465 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:54.861497 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:54.862617 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:54.863476 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:54.864854 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790234094864805 us; error 78 us; skew 500 ppm
Sep 24 07:14:54 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:54.868237 16902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:14:54.869477 16902 webserver.cc:460] Webserver started at http://127.0.98.1:34833/ using document root <none> and password file <none>
I20260924 07:14:54.870240 16902 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:54.870316 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:54.875972 16902 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:54.879184 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:54.881021 16902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:14:54.881181 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "884febdd7e3846e188f2f51b45d38f31"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "c0cd7520f42091323e19eda2f4c49334"
server_key_iv: "c3f60a501d5887b3cf56a759a243b606"
server_key_version: "encryptionkey@0"
I20260924 07:14:54.881886 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:54.894515 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:54.898639 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:54.898874 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:54.899667 16902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:54.901041 16964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:14:54.907438 16902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:14:54.907563 16902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260924 07:14:54.907656 16902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:14:54.910020 16902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:14:54.910081 16902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:14:54.910580 16964 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap starting.
I20260924 07:14:54.922091 16902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35983
I20260924 07:14:54.922397 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35983 every 8 connection(s)
I20260924 07:14:54.923485 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:14:54.933538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 16902
I20260924 07:14:54.934057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39345
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=33213
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:54.934055 16964 log.cc:824] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:54.945024 16964 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:14:54.945497 16964 tablet_bootstrap.cc:492] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Bootstrap complete.
I20260924 07:14:54.946676 16964 ts_tablet_manager.cc:1397] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.013s
W20260924 07:14:54.947751 17074 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:54.925801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:33561)
0924 07:14:54.926365 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:54.926423 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:54.927394 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:54.946333 (+ 18939us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":138,"threads_started":1}
W20260924 07:14:54.948482 17072 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46559 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46559: Client connection negotiation failed: client connection to 127.0.98.62:46559: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:14:54.949535 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:54.949163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:35587)
0924 07:14:54.949336 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:54.949443 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 07:14:54.949872 16964 raft_consensus.cc:348] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:54.950376 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:54.950143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:52731)
0924 07:14:54.950261 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:54.950314 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:14:54.950431 16964 raft_consensus.cc:729] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 884febdd7e3846e188f2f51b45d38f31, State: Initialized, Role: FOLLOWER
W20260924 07:14:54.950546 17072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:14:54.950968 16964 consensus_queue.cc:260] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:54.952112 16964 ts_tablet_manager.cc:1428] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
I20260924 07:14:54.952214 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:54.951963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:54105)
0924 07:14:54.952088 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:54.952147 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:14:54.952922 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:54.952699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:46775)
0924 07:14:54.952813 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:54.952864 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 07:14:55.070008 17076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:55.070382 17076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:55.070441 17076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:55.074952 17076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:55.075064 17076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:55.075134 17076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:14:55.081382 17076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33213
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:55.082860 17076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:55.084180 17076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:55.096406 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:55.096423 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:55.096406 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:55.096850 17076 server_base.cc:1034] running on GCE node
I20260924 07:14:55.097373 17076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:55.098147 17076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:55.099328 17076 hybrid_clock.cc:648] HybridClock initialized: now 1790234095099301 us; error 48 us; skew 500 ppm
Sep 24 07:14:55 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:55.102398 17076 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:14:55.103459 17076 webserver.cc:460] Webserver started at http://127.0.98.2:33213/ using document root <none> and password file <none>
I20260924 07:14:55.104127 17076 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:55.104199 17076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:55.108865 17076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:14:55.111872 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.113588 17076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:14:55.113721 17076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "64933d06e3d24a01bf32a5cb398b5925"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "f5fbc0a9b8ce0a2de6d583841c4b816a"
server_key_iv: "1dc15546b7c1120daa729e5212b78024"
server_key_version: "encryptionkey@0"
I20260924 07:14:55.114368 17076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:55.140131 17076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:55.143765 17076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:55.143936 17076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:55.144625 17076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:55.145514 17076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:55.145563 17076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.145606 17076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:55.145640 17076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.155875 17076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39345
I20260924 07:14:55.155913 17205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39345 every 8 connection(s)
I20260924 07:14:55.157004 17076 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:14:55.162004 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.159791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:45543)
0924 07:14:55.160698 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.160892 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":132,"threads_started":1}
W20260924 07:14:55.162741 17206 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46559 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46559: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
I20260924 07:14:55.162959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17076
I20260924 07:14:55.163496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32971
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46559
--webserver_port=36509
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:55.163640 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.163284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:41463)
0924 07:14:55.163471 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.163542 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:14:55.164485 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.164223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:44235)
0924 07:14:55.164342 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.164412 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 07:14:55.164719 17206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:14:55.282343 17209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:55.282617 17209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:55.282676 17209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:55.286715 17209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:55.286777 17209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:55.286856 17209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:14:55.292331 17209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36509
--tserver_master_addrs=127.0.98.62:46559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:14:55.293406 17209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:55.294413 17209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:55.304584 17214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:55.304634 17217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:55.304651 17209 server_base.cc:1034] running on GCE node
W20260924 07:14:55.304670 17215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:55.305506 17209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:55.306360 17209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:55.307560 17209 hybrid_clock.cc:648] HybridClock initialized: now 1790234095307538 us; error 35 us; skew 500 ppm
Sep 24 07:14:55 dist-test-slave-cnv3 krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:55.311038 17209 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:14:55.312166 17209 webserver.cc:460] Webserver started at http://127.0.98.3:36509/ using document root <none> and password file <none>
I20260924 07:14:55.312737 17209 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:55.312815 17209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:55.317087 17209 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:14:55.319662 17224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.321126 17209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:14:55.321254 17209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "3ed7d3409eed46d69944eb26ba536d17"
format_stamp: "Formatted at 2026-09-24 07:14:12 on dist-test-slave-cnv3"
server_key: "09c1d958b83eb3941f0d84c84a5a2d38"
server_key_iv: "e89aa77ef278a49906f4f448a29eecb0"
server_key_version: "encryptionkey@0"
I20260924 07:14:55.321825 17209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:55.335180 17209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:55.338330 17209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:55.338493 17209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:55.339109 17209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:14:55.340027 17209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:14:55.340078 17209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.340132 17209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:14:55.340160 17209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:55.349804 17209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32971
I20260924 07:14:55.349881 17337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32971 every 8 connection(s)
I20260924 07:14:55.350980 17209 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:14:55.351305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17209
I20260924 07:14:55.356279 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.354024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:42927)
0924 07:14:55.354704 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.354858 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":181,"threads_started":1}
W20260924 07:14:55.356851 17338 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46559 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46559: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
I20260924 07:14:55.357682 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.357349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:50493)
0924 07:14:55.357546 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.357617 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 07:14:55.358467 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.358233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:51725)
0924 07:14:55.358363 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.358412 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:14:55.358697 17338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:14:55.954332 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:55.953794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:37409)
0924 07:14:55.954106 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:55.954170 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
I20260924 07:14:56.165874 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.165410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:38365)
0924 07:14:56.165687 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:56.165751 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
I20260924 07:14:56.249752 17344 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:56.249943 17344 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:56.251063 17344 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
Sep 24 07:14:56 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 07:14:56 dist-test-slave-cnv3 krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 07:14:56.265738 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.251617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39345 (local address 127.0.98.1:58219)
0924 07:14:56.251868 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:56.251882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:56.251976 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:56.254132 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:56.254135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:56.254331 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:56.254829 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:56.254845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.256148 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.256152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:56.257102 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:56.257112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.257772 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:56.257983 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:56.258614 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:56.258630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:56.260485 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:56.262846 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:56.262853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:56.262868 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:56.263331 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:56.263762 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:56.263766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:56.263768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:56.264014 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:56.264829 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:56.264832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:56.265105 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 07:14:56.265269 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 07:14:56.265540 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":210,"spinlock_wait_cycles":50560,"thread_start_us":62,"threads_started":1}
I20260924 07:14:56.265748 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.251320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32971 (local address 127.0.98.1:57909)
0924 07:14:56.251556 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:56.251573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:56.251692 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:56.253752 (+ 2060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:56.253758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:56.254276 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:56.254831 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:56.254845 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.256145 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.256148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:56.257097 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:56.257107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.257772 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:56.257982 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:56.258655 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:56.258666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:56.260845 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:56.262846 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:56.262853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:56.262868 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:56.263331 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:56.263762 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:56.263766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:56.263768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:56.264014 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:56.264817 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:56.264824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:56.265104 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 07:14:56.265267 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 07:14:56.265618 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":175,"spinlock_wait_cycles":18048}
I20260924 07:14:56.266018 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.251574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57909 (local address 127.0.98.3:32971)
0924 07:14:56.251914 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:56.251927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:56.251963 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:14:56.252841 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:56.252851 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:56.253319 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:56.253661 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:56.254994 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.256021 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257210 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.257659 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257801 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:56.261025 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:56.261040 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:56.261057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:56.261091 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:56.262715 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:56.263506 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:56.263511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:56.263518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:56.263664 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:56.264120 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:56.264122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:56.264124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:56.264651 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:56.264814 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:56.265615 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 07:14:56.265831 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":84,"threads_started":1}
I20260924 07:14:56.266458 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.251888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58219 (local address 127.0.98.2:39345)
0924 07:14:56.252172 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:56.252194 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:56.252227 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:14:56.252968 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:56.252975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:56.253318 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:56.253675 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:56.255002 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.256000 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257220 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:56.257658 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:56.257801 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:56.260735 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:56.260740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:56.260753 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:56.260782 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:56.262734 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:56.263505 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:56.263512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:56.263518 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:56.263664 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:56.264113 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:56.264116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:56.264118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:56.264651 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:56.264783 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:56.266027 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0924 07:14:56.266283 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20260924 07:14:56.268193 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
W20260924 07:14:56.268386 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
I20260924 07:14:56.268435 16960 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 884febdd7e3846e188f2f51b45d38f31; no voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925
I20260924 07:14:56.268643 17344 raft_consensus.cc:2726] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:56.359777 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.359286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:59775)
0924 07:14:56.359613 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:56.359669 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":154,"threads_started":1}
I20260924 07:14:56.955780 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:56.955239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:34327)
0924 07:14:56.955584 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:56.955662 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":79,"threads_started":1}
I20260924 07:14:57.167083 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:57.166659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:48541)
0924 07:14:57.166917 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:57.166983 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260924 07:14:57.361160 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:57.360711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:42401)
0924 07:14:57.360991 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:57.361054 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260924 07:14:57.957254 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:57.956747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:55325)
0924 07:14:57.957053 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:57.957112 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":85,"threads_started":1}
I20260924 07:14:58.124402 17353 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:14:58.124608 17353 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:14:58.125016 17353 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
W20260924 07:14:58.125845 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
W20260924 07:14:58.125972 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
I20260924 07:14:58.126021 16960 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 884febdd7e3846e188f2f51b45d38f31; no voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925
I20260924 07:14:58.126194 17353 raft_consensus.cc:2726] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:14:58.168339 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:58.167870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:54065)
0924 07:14:58.168148 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:58.168227 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260924 07:14:58.362509 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:58.362035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:53681)
0924 07:14:58.362342 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:58.362407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260924 07:14:58.958388 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:58.957963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:55497)
0924 07:14:58.958206 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:58.958291 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260924 07:14:59.169668 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:59.169218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:37353)
0924 07:14:59.169501 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:59.169563 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260924 07:14:59.363778 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:59.363313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:46649)
0924 07:14:59.363592 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:59.363644 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":109,"threads_started":1}
I20260924 07:14:59.959769 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:59.959322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:42189)
0924 07:14:59.959618 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:59.959676 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
I20260924 07:15:00.171129 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:00.170635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:38817)
0924 07:15:00.170967 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:00.171031 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260924 07:15:00.283111 17361 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:00.283242 17361 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:15:00.283667 17361 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
W20260924 07:15:00.284530 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
W20260924 07:15:00.284655 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
I20260924 07:15:00.284700 16960 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 884febdd7e3846e188f2f51b45d38f31; no voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925
I20260924 07:15:00.284834 17361 raft_consensus.cc:2726] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:15:00.365136 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:00.364604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:56535)
0924 07:15:00.364976 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:00.365039 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260924 07:15:00.961010 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:00.960545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:36969)
0924 07:15:00.960833 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:00.960904 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
I20260924 07:15:01.172437 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:01.171930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:38109)
0924 07:15:01.172267 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:01.172338 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260924 07:15:01.366485 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:01.366032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:48837)
0924 07:15:01.366317 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:01.366379 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":79,"threads_started":1}
I20260924 07:15:01.962334 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:01.961864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:46305)
0924 07:15:01.962161 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:01.962238 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
I20260924 07:15:02.173811 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:02.173293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:54631)
0924 07:15:02.173627 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:02.173716 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":78,"threads_started":1}
I20260924 07:15:02.367929 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:02.367347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:37659)
0924 07:15:02.367758 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:02.367826 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":82,"threads_started":1}
I20260924 07:15:02.963840 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:02.963336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:34179)
0924 07:15:02.963673 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:02.963743 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260924 07:15:03.175225 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:03.174739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:56717)
0924 07:15:03.175061 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:03.175125 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":80,"threads_started":1}
I20260924 07:15:03.369382 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:03.368838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:33295)
0924 07:15:03.369194 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:03.369275 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260924 07:15:03.775554 17372 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:03.775688 17372 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:15:03.776085 17372 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
W20260924 07:15:03.776928 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
W20260924 07:15:03.777056 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
I20260924 07:15:03.777104 16960 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 884febdd7e3846e188f2f51b45d38f31; no voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925
I20260924 07:15:03.777281 17372 raft_consensus.cc:2726] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:15:03.965170 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:03.964708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:44341)
0924 07:15:03.964993 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:03.965053 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260924 07:15:04.176586 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:04.176074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:57429)
0924 07:15:04.176359 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:04.176481 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260924 07:15:04.370733 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:04.370260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:43263)
0924 07:15:04.370572 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:04.370635 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260924 07:15:04.966689 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:04.966194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:52761)
0924 07:15:04.966528 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:04.966591 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20260924 07:15:05.177852 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:05.177378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:58159)
0924 07:15:05.177698 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:05.177754 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
I20260924 07:15:05.372673 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:05.372143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:34077)
0924 07:15:05.372438 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:05.372574 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260924 07:15:05.967968 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:05.967554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:39485)
0924 07:15:05.967808 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:05.967870 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260924 07:15:06.179075 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:06.178698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:56357)
0924 07:15:06.178928 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:06.178980 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260924 07:15:06.373997 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:06.373569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:37945)
0924 07:15:06.373839 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:06.373902 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260924 07:15:06.969316 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:06.968843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:35393)
0924 07:15:06.969151 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:06.969212 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
I20260924 07:15:07.180310 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:07.179906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:52017)
0924 07:15:07.180144 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:07.180199 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260924 07:15:07.375476 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:07.374933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:60311)
0924 07:15:07.375244 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:07.375310 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":83,"threads_started":1}
I20260924 07:15:07.970709 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:07.970254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:48529)
0924 07:15:07.970544 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:07.970613 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
I20260924 07:15:08.181721 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:08.181271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:46037)
0924 07:15:08.181576 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:08.181626 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":86,"threads_started":1}
I20260924 07:15:08.376901 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:08.376439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:35517)
0924 07:15:08.376738 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:08.376800 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260924 07:15:08.972072 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:08.971629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:33627)
0924 07:15:08.971906 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:08.971973 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":85,"threads_started":1}
I20260924 07:15:09.183223 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:09.182666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:36899)
0924 07:15:09.182956 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:09.183118 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260924 07:15:09.378288 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:09.377799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.3:59235)
0924 07:15:09.378121 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:09.378181 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20260924 07:15:09.973615 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:09.973192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.1:55857)
0924 07:15:09.973456 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:09.973518 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260924 07:15:10.184687 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:10.184235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46559 (local address 127.0.98.2:56185)
0924 07:15:10.184524 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:10.184588 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260924 07:15:10.351022 17393 raft_consensus.cc:482] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:10.351163 17393 raft_consensus.cc:504] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed7d3409eed46d69944eb26ba536d17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32971 } } peers { permanent_uuid: "64933d06e3d24a01bf32a5cb398b5925" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39345 } } peers { permanent_uuid: "884febdd7e3846e188f2f51b45d38f31" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35983 } }
I20260924 07:15:10.351635 17393 leader_election.cc:288] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971), 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345)
W20260924 07:15:10.352415 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 64933d06e3d24a01bf32a5cb398b5925 (127.0.98.2:39345): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
W20260924 07:15:10.352525 16960 leader_election.cc:341] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3ed7d3409eed46d69944eb26ba536d17 (127.0.98.3:32971): Not found: Tablet not found: 30db659c5ba84b3a9b059119da882203
I20260924 07:15:10.352571 16960 leader_election.cc:302] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 884febdd7e3846e188f2f51b45d38f31; no voters: 3ed7d3409eed46d69944eb26ba536d17, 64933d06e3d24a01bf32a5cb398b5925
I20260924 07:15:10.352726 17393 raft_consensus.cc:2726] T 30db659c5ba84b3a9b059119da882203 P 884febdd7e3846e188f2f51b45d38f31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:15:10.363066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 16902
I20260924 07:15:10.368561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17076
I20260924 07:15:10.373769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17209
2026-09-24T07:15:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59374 ms)
[----------] 30 tests from SecurityITest (244251 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:10 dist-test-slave-cnv3 krb5kdc[17396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:10 dist-test-slave-cnv3 krb5kdc[17396](info): set up 2 sockets
Sep 24 07:15:10 dist-test-slave-cnv3 krb5kdc[17396](info): commencing operation
krb5kdc: starting...
W20260924 07:15:11.331284 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:11 dist-test-slave-cnv3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:11.455076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39207
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:11.566344 17409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:11.566645 17409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:11.566687 17409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:11.570834 17409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:11.570914 17409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:11.570931 17409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:11.570948 17409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:11.570963 17409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:11.576613 17409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:11.577776 17409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:11.578821 17409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:11.587679 17417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:11.587724 17409 server_base.cc:1034] running on GCE node
W20260924 07:15:11.587680 17414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:11.587679 17415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:11.588510 17409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:11.589624 17409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:11.590819 17409 hybrid_clock.cc:648] HybridClock initialized: now 1790234111590784 us; error 49 us; skew 500 ppm
Sep 24 07:15:11 dist-test-slave-cnv3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:11.593634 17409 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:11.594535 17409 webserver.cc:460] Webserver started at http://127.0.98.62:44509/ using document root <none> and password file <none>
I20260924 07:15:11.595079 17409 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:11.595135 17409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:11.595295 17409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:11.597359 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "74c61f926263412db44b1433802036cf"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "c30fb08943b5021fae9a7fc5f03097f4"
server_key_iv: "d28408d743b813f6ac0a6e546c0cb190"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.597899 17409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "74c61f926263412db44b1433802036cf"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "c30fb08943b5021fae9a7fc5f03097f4"
server_key_iv: "d28408d743b813f6ac0a6e546c0cb190"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.602005 17409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:15:11.604524 17424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:11.605829 17409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:15:11.605968 17409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "74c61f926263412db44b1433802036cf"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "c30fb08943b5021fae9a7fc5f03097f4"
server_key_iv: "d28408d743b813f6ac0a6e546c0cb190"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.606082 17409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:11.617322 17409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:11.620537 17409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:11.620712 17409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:11.627455 17409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39207
I20260924 07:15:11.627480 17476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39207 every 8 connection(s)
I20260924 07:15:11.628397 17409 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:11.631085 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:11.631613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17409
I20260924 07:15:11.631726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:11.632021 392 external_mini_cluster.cc:1428] Setting key e9259aa3699f283584b055efda1abdde
I20260924 07:15:11.638152 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: Bootstrap starting.
I20260924 07:15:11.640869 17477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: Neither blocks nor log segments found. Creating new log.
Sep 24 07:15:11 dist-test-slave-cnv3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:11.641656 17477 log.cc:824] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:11.643860 17477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: No bootstrap required, opened a new log
I20260924 07:15:11.646354 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:11.633394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.0.98.62:39207)
0924 07:15:11.633783 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:11.633797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:11.633830 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:15:11.634490 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:11.634505 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:11.634819 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:11.635091 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:11.636057 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.637089 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.638308 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.638843 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.638968 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:11.641701 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:11.641708 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:11.641724 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:11.641758 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:11.643473 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:11.643931 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:11.643937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:11.643943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:11.644063 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:11.644436 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:11.644439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:11.644440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:11.644849 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:11.645006 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:11.645314 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 07:15:11.645617 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260924 07:15:11.647847 17477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } }
I20260924 07:15:11.648118 17477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:11.648176 17477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c61f926263412db44b1433802036cf, State: Initialized, Role: FOLLOWER
I20260924 07:15:11.648605 17477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } }
I20260924 07:15:11.648736 17477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:11.648806 17477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:11.648901 17477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:11.649814 17477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } }
I20260924 07:15:11.650136 17477 leader_election.cc:302] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74c61f926263412db44b1433802036cf; no voters:
I20260924 07:15:11.650455 17477 leader_election.cc:288] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:11.650597 17482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:11.650856 17482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [term 1 LEADER]: Becoming Leader. State: Replica: 74c61f926263412db44b1433802036cf, State: Running, Role: LEADER
I20260924 07:15:11.651221 17482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } }
I20260924 07:15:11.651698 17477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:11.653013 17483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74c61f926263412db44b1433802036cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } } }
I20260924 07:15:11.653064 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74c61f926263412db44b1433802036cf. Latest consensus state: current_term: 1 leader_uuid: "74c61f926263412db44b1433802036cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74c61f926263412db44b1433802036cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39207 } } }
I20260924 07:15:11.653151 17483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:11.653177 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:11.653522 17494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:11.656500 17494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:11.662029 17494 catalog_manager.cc:1347] Generated new cluster ID: e6a3905ff0fd47679382532f891d5863
I20260924 07:15:11.662118 17494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:11.697470 17494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:11.698457 17494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:11.702780 17494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: Generated new TSK 0
I20260924 07:15:11.703518 17494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:15:11.734742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:11.846351 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:11.846652 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:11.846695 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:11.850819 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:11.850885 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:11.850945 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:11.856878 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:11.857963 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:11.858903 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:11.868871 17511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:11.868906 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:11.868899 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:11.869242 17503 server_base.cc:1034] running on GCE node
I20260924 07:15:11.869663 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:11.870414 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:11.871587 17503 hybrid_clock.cc:648] HybridClock initialized: now 1790234111871555 us; error 46 us; skew 500 ppm
Sep 24 07:15:11 dist-test-slave-cnv3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:11.874439 17503 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:15:11.875411 17503 webserver.cc:460] Webserver started at http://127.0.98.1:45119/ using document root <none> and password file <none>
I20260924 07:15:11.875953 17503 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:11.876048 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:11.876245 17503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:11.878301 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "b2d58b7140984232b8cac97ea85b5595"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "bf2fc1e7a2d80b0628bab752682256de"
server_key_iv: "372a47ab42e97929de6a97ce0bf8d45c"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.878849 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b2d58b7140984232b8cac97ea85b5595"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "bf2fc1e7a2d80b0628bab752682256de"
server_key_iv: "372a47ab42e97929de6a97ce0bf8d45c"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.882786 17503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:11.885488 17518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:11.886765 17503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:15:11.886896 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "b2d58b7140984232b8cac97ea85b5595"
format_stamp: "Formatted at 2026-09-24 07:15:11 on dist-test-slave-cnv3"
server_key: "bf2fc1e7a2d80b0628bab752682256de"
server_key_iv: "372a47ab42e97929de6a97ce0bf8d45c"
server_key_version: "encryptionkey@0"
I20260924 07:15:11.887007 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:11.933908 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:11.937222 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:11.937412 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:11.938009 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:11.938827 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:11.938874 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:11.938925 17503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:11.938956 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:11.947690 17503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39153
I20260924 07:15:11.947749 17631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39153 every 8 connection(s)
I20260924 07:15:11.948604 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:11.951834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17503
I20260924 07:15:11.951967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:11.952268 392 external_mini_cluster.cc:1428] Setting key 9505ebcd88f2212c02909d7842087cf4
Sep 24 07:15:11 dist-test-slave-cnv3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:15:11.964079 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:11.950514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58001 (local address 127.0.98.62:39207)
0924 07:15:11.950697 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:11.950701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:11.951588 (+ 887us) server_negotiation.cc:407] Connection header received
0924 07:15:11.952967 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:11.952970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:11.953016 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:11.953097 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:11.954355 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.955096 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.956793 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.957044 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.957124 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:11.960024 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:11.960029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:11.960032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:11.960064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:11.961804 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:11.962394 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:11.962399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:11.962401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:11.962493 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:11.962801 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:11.962804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:11.962805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:11.963013 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:11.963096 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:11.963776 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 07:15:11.963929 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:15:11.965034 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:11.950854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39207 (local address 127.0.98.1:58001)
0924 07:15:11.951432 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:11.951484 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:11.952775 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:11.953264 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:11.953272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:11.953741 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:11.954216 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:11.954229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.955263 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.955267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:11.956679 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:11.956690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:11.957189 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:11.957193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:11.957386 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:11.958034 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:11.958057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:11.959825 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:11.961945 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:11.961950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:11.961964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:11.962273 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:11.962567 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:11.962569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:11.962570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:11.962711 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:11.963125 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:11.963133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:11.963537 (+ 404us) client_negotiation.cc:769] Sending connection context
0924 07:15:11.963707 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 07:15:11.964000 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260924 07:15:11.966367 17632 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39207
I20260924 07:15:11.966611 17632 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:11.967137 17632 heartbeater.cc:511] Master 127.0.98.62:39207 requested a full tablet report, sending...
I20260924 07:15:11.968724 17441 ts_manager.cc:194] Registered new tserver with Master: b2d58b7140984232b8cac97ea85b5595 (127.0.98.1:39153)
I20260924 07:15:11.969868 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:15:11.983671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:12.095999 17637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:12.096294 17637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:12.096338 17637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:12.100481 17637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:12.100549 17637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:12.100610 17637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:12.106158 17637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:12.107250 17637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:12.108278 17637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:12.118093 17642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:12.118084 17643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:12.118269 17637 server_base.cc:1034] running on GCE node
W20260924 07:15:12.118084 17645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:12.118964 17637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:12.119704 17637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:12.120886 17637 hybrid_clock.cc:648] HybridClock initialized: now 1790234112120867 us; error 36 us; skew 500 ppm
Sep 24 07:15:12 dist-test-slave-cnv3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:12.123793 17637 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:15:12.124697 17637 webserver.cc:460] Webserver started at http://127.0.98.2:34933/ using document root <none> and password file <none>
I20260924 07:15:12.125241 17637 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:12.125303 17637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:12.125521 17637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:12.127614 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "fc5fd3f37ee54f239d7f702b8d634b89"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "3b12344437453241946aa9bdbbcd1c9e"
server_key_iv: "ffd9f83c30d258cc1e324e8ea3d68a79"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.128146 17637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fc5fd3f37ee54f239d7f702b8d634b89"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "3b12344437453241946aa9bdbbcd1c9e"
server_key_iv: "ffd9f83c30d258cc1e324e8ea3d68a79"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.131969 17637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:15:12.134783 17652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.136271 17637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:12.136440 17637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "fc5fd3f37ee54f239d7f702b8d634b89"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "3b12344437453241946aa9bdbbcd1c9e"
server_key_iv: "ffd9f83c30d258cc1e324e8ea3d68a79"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.136566 17637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:12.158914 17637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:12.162264 17637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:12.162436 17637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:12.163030 17637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:12.163877 17637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:12.163925 17637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.163964 17637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:12.163990 17637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.172418 17637 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33309
I20260924 07:15:12.172492 17765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33309 every 8 connection(s)
I20260924 07:15:12.173301 17637 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:12.180568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17637
I20260924 07:15:12.180665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:12.180948 392 external_mini_cluster.cc:1428] Setting key 11381e6e1d6f186bbe40839791e736b4
Sep 24 07:15:12 dist-test-slave-cnv3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:12.187633 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.175181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45009 (local address 127.0.98.62:39207)
0924 07:15:12.175320 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.175324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.176179 (+ 855us) server_negotiation.cc:407] Connection header received
0924 07:15:12.177422 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.177426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.177495 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.177564 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:12.178811 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.179344 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.180558 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.180819 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.180907 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.183722 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:12.183726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:12.183728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:12.183754 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:12.185263 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.185814 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.185818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.185820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.185907 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.186292 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.186295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.186297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.186564 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:12.186670 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.187260 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.187433 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:15:12.188522 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.175590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39207 (local address 127.0.98.2:45009)
0924 07:15:12.176035 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.176078 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.177215 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.177741 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.177748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.178150 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:12.178659 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.178680 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.179487 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.179490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.180441 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.180450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.180984 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.180989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.181163 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.181797 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:12.181814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:12.183541 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:12.185420 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:12.185427 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:12.185442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:12.185723 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:12.186023 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:12.186025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:12.186027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:12.186194 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:12.186692 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:12.186698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:12.186994 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.187184 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.187595 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260924 07:15:12.189800 17766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39207
I20260924 07:15:12.190057 17766 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:15:12.190734 17766 heartbeater.cc:511] Master 127.0.98.62:39207 requested a full tablet report, sending...
I20260924 07:15:12.191947 17441 ts_manager.cc:194] Registered new tserver with Master: fc5fd3f37ee54f239d7f702b8d634b89 (127.0.98.2:33309)
I20260924 07:15:12.192610 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:15:12.212342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:12.323804 17771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:12.324087 17771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:12.324134 17771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:12.328317 17771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:12.328389 17771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:12.328459 17771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:12.333961 17771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:12.335078 17771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:12.336099 17771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:12.345883 17771 server_base.cc:1034] running on GCE node
W20260924 07:15:12.346030 17779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:12.346091 17776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:12.345839 17777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:12.346565 17771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:12.347235 17771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:12.348413 17771 hybrid_clock.cc:648] HybridClock initialized: now 1790234112348392 us; error 41 us; skew 500 ppm
Sep 24 07:15:12 dist-test-slave-cnv3 krb5kdc[17396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:12.351102 17771 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:15:12.352082 17771 webserver.cc:460] Webserver started at http://127.0.98.3:44575/ using document root <none> and password file <none>
I20260924 07:15:12.352682 17771 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:12.352746 17771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:12.352924 17771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:12.355029 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "aaf32872274d44c086ea527f76d7d6a1"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "cba21dd39200a92670309a02bfcf5629"
server_key_iv: "223dea0f1c66b8c3c985498dd7f96cdd"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.355595 17771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aaf32872274d44c086ea527f76d7d6a1"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "cba21dd39200a92670309a02bfcf5629"
server_key_iv: "223dea0f1c66b8c3c985498dd7f96cdd"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.359527 17771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:15:12.361936 17786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.363294 17771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:15:12.363458 17771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "aaf32872274d44c086ea527f76d7d6a1"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "cba21dd39200a92670309a02bfcf5629"
server_key_iv: "223dea0f1c66b8c3c985498dd7f96cdd"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.363574 17771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:12.375130 17771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:12.378278 17771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:12.378445 17771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:12.379047 17771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:12.379897 17771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:12.379944 17771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.379983 17771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:12.379999 17771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.388993 17771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44583
I20260924 07:15:12.389019 17899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44583 every 8 connection(s)
I20260924 07:15:12.389926 17771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:12.398862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17771
I20260924 07:15:12.399013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:12.399299 392 external_mini_cluster.cc:1428] Setting key e18837f9b82a830c5a1ab02895e57c03
Sep 24 07:15:12 dist-test-slave-cnv3 krb5kdc[17396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:12.403779 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.391944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57677 (local address 127.0.98.62:39207)
0924 07:15:12.392088 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.392093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.393004 (+ 911us) server_negotiation.cc:407] Connection header received
0924 07:15:12.393851 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.393854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.393906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.393987 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:12.395155 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.395783 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.397236 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.397510 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.397600 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.400363 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:12.400367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:12.400370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:12.400410 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:12.401749 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.402237 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.402240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.402242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.402335 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.402658 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.402661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.402663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.402855 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:12.402950 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.403488 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.403617 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:15:12.404692 17902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.392349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39207 (local address 127.0.98.3:57677)
0924 07:15:12.392808 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.392856 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.393626 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.394128 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.394135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.394484 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:12.395012 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.395024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.395926 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.395929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.397110 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.397121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.397652 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.397655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.397798 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.398403 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:12.398419 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:12.400193 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:12.401860 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:12.401866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:12.401888 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:12.402144 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:12.402423 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:12.402426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:12.402429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:12.402574 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:12.402951 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:12.402959 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:12.403238 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.403447 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.403822 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260924 07:15:12.405910 17900 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39207
I20260924 07:15:12.406147 17900 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:12.406641 17900 heartbeater.cc:511] Master 127.0.98.62:39207 requested a full tablet report, sending...
I20260924 07:15:12.407681 17441 ts_manager.cc:194] Registered new tserver with Master: aaf32872274d44c086ea527f76d7d6a1 (127.0.98.3:44583)
I20260924 07:15:12.408250 17441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57677
I20260924 07:15:12.413511 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:12.422494 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.415344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.0.98.62:39207)
0924 07:15:12.415532 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.415537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.415639 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:15:12.415737 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.415739 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.415786 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.415847 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:12.416486 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.416997 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.417905 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.418122 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.418198 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.419491 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:12.419495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:12.419497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:12.419521 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:12.420748 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.421178 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.421181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.421182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.421260 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:12.421578 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:12.421581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:12.421582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:12.421829 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:12.421945 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.422208 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.422329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 07:15:12.425688 17441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:12.428813 17441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:12.444031 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.437868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.0.98.1:39153)
0924 07:15:12.438949 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.438971 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.438994 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:15:12.439071 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.439077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.439287 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.439943 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:12.440121 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.440943 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.442524 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443273 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.443581 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.443757 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.443865 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":96,"threads_started":1}
I20260924 07:15:12.444033 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.437715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39153 (local address 127.0.0.1:41734)
0924 07:15:12.438471 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.438527 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.438664 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.439491 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.439497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.439530 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:12.439808 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.439816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.441091 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.441095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.442389 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.442399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443423 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.443547 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.443592 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.443715 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.443852 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":75,"spinlock_wait_cycles":7680,"thread_start_us":71,"threads_started":1}
I20260924 07:15:12.444321 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.438485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33309 (local address 127.0.0.1:39920)
0924 07:15:12.439321 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.439335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.439455 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.439942 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.439944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.439970 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:12.440047 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.440053 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.441447 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.441451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.442705 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.442715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443921 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.444023 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.444046 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.444117 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.444198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":46,"thread_start_us":98,"threads_started":1}
I20260924 07:15:12.444823 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.438915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.0.98.2:33309)
0924 07:15:12.439249 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.439259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.439446 (+ 187us) server_negotiation.cc:407] Connection header received
0924 07:15:12.439608 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.439614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.439799 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.439934 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:12.440159 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.441290 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.442838 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443645 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443734 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.443858 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.444558 (+ 700us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.444658 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260924 07:15:12.445362 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.437796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44583 (local address 127.0.0.1:42626)
0924 07:15:12.438472 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.438530 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.438663 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.439504 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.439506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.439531 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:12.439808 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.439816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.441447 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.441451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.442910 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.442917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443998 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.444001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.444086 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.444107 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.445158 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.445233 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":286,"spinlock_wait_cycles":19840,"thread_start_us":100,"threads_started":1}
I20260924 07:15:12.445844 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.438020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.98.3:44583)
0924 07:15:12.438947 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.438958 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.438982 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:15:12.439072 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.439077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.439287 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.439469 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:12.439928 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.441312 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443018 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.443723 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.443811 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.443939 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.445598 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.445702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":88,"threads_started":1}
I20260924 07:15:12.447104 17565 tablet_service.cc:1467] Processing CreateTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 (DEFAULT_TABLE table=test-table [id=8a1592d7797b478eba0a4bfd6784ec10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:12.447104 17700 tablet_service.cc:1467] Processing CreateTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 (DEFAULT_TABLE table=test-table [id=8a1592d7797b478eba0a4bfd6784ec10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:12.448094 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d390f852c5b4a81bf9513c51f78f912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:12.448094 17565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d390f852c5b4a81bf9513c51f78f912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:12.454528 17917 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: Bootstrap starting.
I20260924 07:15:12.454557 17834 tablet_service.cc:1467] Processing CreateTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 (DEFAULT_TABLE table=test-table [id=8a1592d7797b478eba0a4bfd6784ec10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:12.456067 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d390f852c5b4a81bf9513c51f78f912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:12.456856 17917 tablet_bootstrap.cc:654] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:12.457896 17917 log.cc:824] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:12.459913 17917 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: No bootstrap required, opened a new log
I20260924 07:15:12.460120 17917 ts_tablet_manager.cc:1397] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:15:12.461858 17918 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Bootstrap starting.
I20260924 07:15:12.462744 17920 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Bootstrap starting.
I20260924 07:15:12.463671 17917 raft_consensus.cc:348] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.463973 17917 raft_consensus.cc:374] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:12.464067 17917 raft_consensus.cc:729] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d58b7140984232b8cac97ea85b5595, State: Initialized, Role: FOLLOWER
I20260924 07:15:12.464486 17918 tablet_bootstrap.cc:654] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:12.464572 17917 consensus_queue.cc:260] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.465058 17920 tablet_bootstrap.cc:654] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:12.465327 17632 heartbeater.cc:503] Master 127.0.98.62:39207 was elected leader, sending a full tablet report...
I20260924 07:15:12.465747 17917 ts_tablet_manager.cc:1428] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:15:12.466346 17920 log.cc:824] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:12.467015 17918 log.cc:824] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:12.468734 17920 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: No bootstrap required, opened a new log
I20260924 07:15:12.468931 17918 tablet_bootstrap.cc:492] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: No bootstrap required, opened a new log
I20260924 07:15:12.468986 17920 ts_tablet_manager.cc:1397] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:15:12.469132 17918 ts_tablet_manager.cc:1397] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260924 07:15:12.472425 17920 raft_consensus.cc:348] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.472499 17918 raft_consensus.cc:348] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.472728 17918 raft_consensus.cc:374] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:12.472741 17920 raft_consensus.cc:374] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:12.472764 17918 raft_consensus.cc:729] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc5fd3f37ee54f239d7f702b8d634b89, State: Initialized, Role: FOLLOWER
I20260924 07:15:12.472807 17920 raft_consensus.cc:729] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf32872274d44c086ea527f76d7d6a1, State: Initialized, Role: FOLLOWER
I20260924 07:15:12.473233 17920 consensus_queue.cc:260] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.473233 17918 consensus_queue.cc:260] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.473937 17900 heartbeater.cc:503] Master 127.0.98.62:39207 was elected leader, sending a full tablet report...
I20260924 07:15:12.474117 17766 heartbeater.cc:503] Master 127.0.98.62:39207 was elected leader, sending a full tablet report...
I20260924 07:15:12.474270 17920 ts_tablet_manager.cc:1428] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:15:12.474272 17918 ts_tablet_manager.cc:1428] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260924 07:15:12.586800 17925 raft_consensus.cc:482] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:12.587033 17925 raft_consensus.cc:504] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.588263 17925 leader_election.cc:288] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d58b7140984232b8cac97ea85b5595 (127.0.98.1:39153), aaf32872274d44c086ea527f76d7d6a1 (127.0.98.3:44583)
I20260924 07:15:12.593441 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.588676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50591 (local address 127.0.98.3:44583)
0924 07:15:12.588816 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.588820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.588839 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:15:12.588955 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.588959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.589016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.589105 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:12.589574 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.590381 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592161 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.592790 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592910 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.592976 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.593235 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.593305 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:15:12.593441 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.588748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56813 (local address 127.0.98.1:39153)
0924 07:15:12.588880 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.588884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.589286 (+ 402us) server_negotiation.cc:407] Connection header received
0924 07:15:12.589385 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.589388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.589425 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.589482 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:12.589723 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.590461 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592161 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.592790 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592910 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.592976 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.593231 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.593305 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:15:12.593479 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.588581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44583 (local address 127.0.98.2:50591)
0924 07:15:12.588731 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.588747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.588854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.589149 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.589153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.589173 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:12.589297 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.589303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.590513 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.590517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.592048 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.592060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.592935 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.593095 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.593125 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.593203 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.593304 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":81}
I20260924 07:15:12.593487 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.588599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39153 (local address 127.0.98.2:56813)
0924 07:15:12.589192 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.589206 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:12.589302 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:12.589495 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:12.589497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:12.589512 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:12.589608 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.589612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.590570 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.590572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.592048 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:12.592060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.592935 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.592940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:12.593075 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.593135 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 07:15:12.593207 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:15:12.593304 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":317,"thread_start_us":163,"threads_started":1}
I20260924 07:15:12.594208 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d390f852c5b4a81bf9513c51f78f912" candidate_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf32872274d44c086ea527f76d7d6a1" is_pre_election: true
I20260924 07:15:12.594519 17854 raft_consensus.cc:2454] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5fd3f37ee54f239d7f702b8d634b89 in term 0.
I20260924 07:15:12.594986 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d390f852c5b4a81bf9513c51f78f912" candidate_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d58b7140984232b8cac97ea85b5595" is_pre_election: true
I20260924 07:15:12.595082 17655 leader_election.cc:302] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaf32872274d44c086ea527f76d7d6a1, fc5fd3f37ee54f239d7f702b8d634b89; no voters:
I20260924 07:15:12.595263 17586 raft_consensus.cc:2454] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc5fd3f37ee54f239d7f702b8d634b89 in term 0.
I20260924 07:15:12.595432 17925 raft_consensus.cc:2781] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:12.595523 17925 raft_consensus.cc:482] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:12.595570 17925 raft_consensus.cc:3037] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:12.596645 17925 raft_consensus.cc:504] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.596993 17925 leader_election.cc:288] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [CANDIDATE]: Term 1 election: Requested vote from peers b2d58b7140984232b8cac97ea85b5595 (127.0.98.1:39153), aaf32872274d44c086ea527f76d7d6a1 (127.0.98.3:44583)
I20260924 07:15:12.597419 17586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d390f852c5b4a81bf9513c51f78f912" candidate_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d58b7140984232b8cac97ea85b5595"
I20260924 07:15:12.597414 17854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d390f852c5b4a81bf9513c51f78f912" candidate_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf32872274d44c086ea527f76d7d6a1"
I20260924 07:15:12.597558 17586 raft_consensus.cc:3037] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:12.597618 17854 raft_consensus.cc:3037] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:12.598883 17586 raft_consensus.cc:2454] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5fd3f37ee54f239d7f702b8d634b89 in term 1.
I20260924 07:15:12.598883 17854 raft_consensus.cc:2454] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc5fd3f37ee54f239d7f702b8d634b89 in term 1.
I20260924 07:15:12.599222 17655 leader_election.cc:302] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaf32872274d44c086ea527f76d7d6a1, fc5fd3f37ee54f239d7f702b8d634b89; no voters:
I20260924 07:15:12.599509 17925 raft_consensus.cc:2781] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:12.599753 17925 raft_consensus.cc:686] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 1 LEADER]: Becoming Leader. State: Replica: fc5fd3f37ee54f239d7f702b8d634b89, State: Running, Role: LEADER
I20260924 07:15:12.600129 17925 consensus_queue.cc:237] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } }
I20260924 07:15:12.603350 17441 catalog_manager.cc:5495] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc5fd3f37ee54f239d7f702b8d634b89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc5fd3f37ee54f239d7f702b8d634b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33309 } health_report { overall_health: HEALTHY } } }
I20260924 07:15:12.637439 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.633450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.98.2:33309)
0924 07:15:12.633589 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.633592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.633608 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:12.633739 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.633741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.633798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.633898 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:12.634161 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.634675 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.635919 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.636161 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.636257 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.636590 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:12.637030 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:12.637144 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.637218 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.637275 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 07:15:12.640923 17901 tablet.cc:2369] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:12.643694 17854 raft_consensus.cc:1261] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 1 FOLLOWER]: Refusing update from remote peer fc5fd3f37ee54f239d7f702b8d634b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:12.643694 17586 raft_consensus.cc:1261] T 7d390f852c5b4a81bf9513c51f78f912 P b2d58b7140984232b8cac97ea85b5595 [term 1 FOLLOWER]: Refusing update from remote peer fc5fd3f37ee54f239d7f702b8d634b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:12.644472 17927 consensus_queue.cc:1035] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d58b7140984232b8cac97ea85b5595" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:12.644662 17925 consensus_queue.cc:1035] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf32872274d44c086ea527f76d7d6a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:12.655211 17931 mvcc.cc:205] Tried to move back new op lower bound from 7332798925380554752 to 7332798925212491776. Current Snapshot: MvccSnapshot[applied={T|T < 7332798925380554752 or (T in {7332798925380554752})}]
I20260924 07:15:12.660521 17439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:12.660688 17439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:12.663076 17439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74c61f926263412db44b1433802036cf: Sending DeleteTablet for 3 replicas of tablet 7d390f852c5b4a81bf9513c51f78f912
I20260924 07:15:12.664049 17565 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41734
I20260924 07:15:12.664048 17834 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42626
I20260924 07:15:12.664525 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17503
I20260924 07:15:12.664796 17700 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d390f852c5b4a81bf9513c51f78f912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260924 07:15:12.665302 17940 tablet_replica.cc:330] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: stopping tablet replica
I20260924 07:15:12.665666 17940 raft_consensus.cc:2229] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:12.666015 17940 raft_consensus.cc:2258] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:12.666563 17942 tablet_replica.cc:330] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: stopping tablet replica
I20260924 07:15:12.666899 17942 raft_consensus.cc:2229] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:15:12.667335 17942 raft_consensus.cc:2258] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:12.668548 17940 ts_tablet_manager.cc:1905] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:12.668798 17942 ts_tablet_manager.cc:1905] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:12.671415 17940 ts_tablet_manager.cc:1918] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:12.671618 17940 log.cc:1196] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/7d390f852c5b4a81bf9513c51f78f912
I20260924 07:15:12.671694 17942 ts_tablet_manager.cc:1918] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:12.671891 17942 log.cc:1196] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/7d390f852c5b4a81bf9513c51f78f912
I20260924 07:15:12.671983 17940 ts_tablet_manager.cc:1939] T 7d390f852c5b4a81bf9513c51f78f912 P aaf32872274d44c086ea527f76d7d6a1: Deleting consensus metadata
I20260924 07:15:12.672248 17942 ts_tablet_manager.cc:1939] T 7d390f852c5b4a81bf9513c51f78f912 P fc5fd3f37ee54f239d7f702b8d634b89: Deleting consensus metadata
W20260924 07:15:12.673156 17425 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39153 (error 108)
I20260924 07:15:12.673390 17427 catalog_manager.cc:4841] TS aaf32872274d44c086ea527f76d7d6a1 (127.0.98.3:44583): tablet 7d390f852c5b4a81bf9513c51f78f912 (table test-table [id=8a1592d7797b478eba0a4bfd6784ec10]) successfully deleted
I20260924 07:15:12.673396 17428 catalog_manager.cc:4841] TS fc5fd3f37ee54f239d7f702b8d634b89 (127.0.98.2:33309): tablet 7d390f852c5b4a81bf9513c51f78f912 (table test-table [id=8a1592d7797b478eba0a4bfd6784ec10]) successfully deleted
I20260924 07:15:12.673990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17637
I20260924 07:15:12.676353 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.676024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39153 (local address 127.0.0.1:41746)
0924 07:15:12.676164 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:12.676249 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 07:15:12.676679 17425 catalog_manager.cc:4584] TS b2d58b7140984232b8cac97ea85b5595 (127.0.98.1:39153): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d390f852c5b4a81bf9513c51f78f912: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39153: connect: Connection refused (error 111)
I20260924 07:15:12.679409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17771
I20260924 07:15:12.684482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17409
2026-09-24T07:15:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2310 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T07:15:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:12Z Disabled control of system clock
I20260924 07:15:12.713549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45775
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:12.823882 17947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:12.824179 17947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:12.824215 17947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:12.828351 17947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:12.828430 17947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:12.828446 17947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:12.828462 17947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:12.828476 17947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:12.833635 17947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45775
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:12.834777 17947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:12.836228 17947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:12.844578 17952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:12.844648 17955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:12.844573 17953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:12.845036 17947 server_base.cc:1034] running on GCE node
I20260924 07:15:12.845611 17947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:12.846763 17947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:12.847926 17947 hybrid_clock.cc:648] HybridClock initialized: now 1790234112847914 us; error 28 us; skew 500 ppm
I20260924 07:15:12.849918 17947 webserver.cc:460] Webserver started at http://127.0.98.62:45319/ using document root <none> and password file <none>
I20260924 07:15:12.850500 17947 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:12.850623 17947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:12.850855 17947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:12.853024 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "b4f077802f6a47ddb15ddd550decf7e4"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "bad6940b26c27bab61d00d0aebc38e30"
server_key_iv: "5615033187d1033e67637eebe99558ca"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.853574 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4f077802f6a47ddb15ddd550decf7e4"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "bad6940b26c27bab61d00d0aebc38e30"
server_key_iv: "5615033187d1033e67637eebe99558ca"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.857281 17947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 07:15:12.860059 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:12.861371 17947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:12.861496 17947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "b4f077802f6a47ddb15ddd550decf7e4"
format_stamp: "Formatted at 2026-09-24 07:15:12 on dist-test-slave-cnv3"
server_key: "bad6940b26c27bab61d00d0aebc38e30"
server_key_iv: "5615033187d1033e67637eebe99558ca"
server_key_version: "encryptionkey@0"
I20260924 07:15:12.861666 17947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:12.878285 17947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:12.879012 17947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:12.879196 17947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:12.885921 17947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45775
I20260924 07:15:12.885946 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45775 every 8 connection(s)
I20260924 07:15:12.886893 17947 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:12.889731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 17947
I20260924 07:15:12.889868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:12.890173 392 external_mini_cluster.cc:1428] Setting key 90fcbe210ce851814bfa2720c1e9a41a
I20260924 07:15:12.890455 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:12.897985 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: Bootstrap starting.
I20260924 07:15:12.900640 18014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:12.900898 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:12.891624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.0.98.62:45775)
0924 07:15:12.892334 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:12.892353 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:12.892395 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:15:12.893345 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:12.893369 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:12.893911 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:12.894310 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:12.894757 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.896211 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.897112 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:12.897791 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:12.897964 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:12.898791 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:12.898797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:12.898812 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:12.899317 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:12.899414 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:12.899443 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:12.899535 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:12.899687 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:15:12.899959 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260924 07:15:12.901476 18014 log.cc:824] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:12.903508 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: No bootstrap required, opened a new log
I20260924 07:15:12.907177 18014 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } }
I20260924 07:15:12.907472 18014 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:12.907543 18014 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f077802f6a47ddb15ddd550decf7e4, State: Initialized, Role: FOLLOWER
I20260924 07:15:12.907946 18014 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } }
I20260924 07:15:12.908087 18014 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:12.908151 18014 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:12.908246 18014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:12.909161 18014 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } }
I20260924 07:15:12.909490 18014 leader_election.cc:302] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f077802f6a47ddb15ddd550decf7e4; no voters:
I20260924 07:15:12.909794 18014 leader_election.cc:288] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:12.909907 18019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:12.910137 18019 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [term 1 LEADER]: Becoming Leader. State: Replica: b4f077802f6a47ddb15ddd550decf7e4, State: Running, Role: LEADER
I20260924 07:15:12.910468 18019 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } }
I20260924 07:15:12.911005 18014 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:12.912084 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f077802f6a47ddb15ddd550decf7e4. Latest consensus state: current_term: 1 leader_uuid: "b4f077802f6a47ddb15ddd550decf7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } }
I20260924 07:15:12.912211 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:12.912453 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f077802f6a47ddb15ddd550decf7e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f077802f6a47ddb15ddd550decf7e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45775 } } }
I20260924 07:15:12.912542 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:12.912657 18028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:12.915856 18028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:12.921603 18028 catalog_manager.cc:1347] Generated new cluster ID: ca9309c98b644c4fb4c5b4b0154cd9c6
I20260924 07:15:12.921679 18028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:12.933447 18028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:12.934409 18028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:12.946065 18028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: Generated new TSK 0
I20260924 07:15:12.946920 18028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:12.958971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:13.072438 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:13.072718 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:13.072772 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:13.076977 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:13.077091 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:13.077173 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:13.082340 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:13.083540 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:13.084955 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:13.094944 18038 server_base.cc:1034] running on GCE node
W20260924 07:15:13.094900 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.094905 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.094900 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:13.095706 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:13.096369 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:13.097543 18038 hybrid_clock.cc:648] HybridClock initialized: now 1790234113097500 us; error 57 us; skew 500 ppm
I20260924 07:15:13.099359 18038 webserver.cc:460] Webserver started at http://127.0.98.1:42205/ using document root <none> and password file <none>
I20260924 07:15:13.099957 18038 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:13.100035 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:13.100237 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:13.102246 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "3125a31f904d4201939d65257979043e"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "12215233fa849f9e196f8886f136f587"
server_key_iv: "29f22aef222eead23dbdf86e5bd40f9e"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.102769 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3125a31f904d4201939d65257979043e"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "12215233fa849f9e196f8886f136f587"
server_key_iv: "29f22aef222eead23dbdf86e5bd40f9e"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.106529 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:15:13.109104 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.110374 18038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:13.110500 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "3125a31f904d4201939d65257979043e"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "12215233fa849f9e196f8886f136f587"
server_key_iv: "29f22aef222eead23dbdf86e5bd40f9e"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.110610 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:13.124186 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:13.124940 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:13.125124 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:13.125703 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:13.126652 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:13.126699 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.126752 18038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:13.126780 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.135452 18038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33465
I20260924 07:15:13.135691 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33465 every 8 connection(s)
I20260924 07:15:13.136405 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:13.145468 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.138173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40375 (local address 127.0.98.62:45775)
0924 07:15:13.138336 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.138341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.139356 (+ 1015us) server_negotiation.cc:407] Connection header received
0924 07:15:13.140750 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.140753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.140804 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.140880 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:13.141749 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.142332 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.143490 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.143756 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.143843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.144982 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:13.144985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:13.144987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:13.145001 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:13.145020 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:13.145028 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:13.145110 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.145272 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.145346 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:15:13.145998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18038
I20260924 07:15:13.146122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:13.146405 392 external_mini_cluster.cc:1428] Setting key 380b7819d0aeb5b43345a2acdb1cdfad
I20260924 07:15:13.146471 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.138572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.98.1:40375)
0924 07:15:13.139213 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.139255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.140538 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.141038 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.141044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.141127 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:13.141626 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.141639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.142452 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.142455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.143348 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.143358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.143920 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.143923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.144088 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.144717 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:13.144719 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:13.144760 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:13.144827 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:13.145119 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:13.145124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:13.145128 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.145218 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.145446 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260924 07:15:13.147958 18166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45775
I20260924 07:15:13.148231 18166 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:13.148404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:13.148787 18166 heartbeater.cc:511] Master 127.0.98.62:45775 requested a full tablet report, sending...
I20260924 07:15:13.150354 17976 ts_manager.cc:194] Registered new tserver with Master: 3125a31f904d4201939d65257979043e (127.0.98.1:33465)
I20260924 07:15:13.151902 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40375
W20260924 07:15:13.264290 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:13.264580 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:13.264626 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:13.268855 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:13.268927 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:13.268985 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:13.273888 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:13.274968 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:13.276364 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:13.285964 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.285992 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.285964 18177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:13.286257 18169 server_base.cc:1034] running on GCE node
I20260924 07:15:13.286711 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:13.287416 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:13.288625 18169 hybrid_clock.cc:648] HybridClock initialized: now 1790234113288567 us; error 69 us; skew 500 ppm
I20260924 07:15:13.290712 18169 webserver.cc:460] Webserver started at http://127.0.98.2:33673/ using document root <none> and password file <none>
I20260924 07:15:13.291491 18169 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:13.291569 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:13.291742 18169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:13.293838 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "86730a9c0a684f52809759c01515bba2"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b1afe2fc66662457684900358fed9098"
server_key_iv: "395afc2ca9a20f20106c216a36ac1c04"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.294502 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86730a9c0a684f52809759c01515bba2"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b1afe2fc66662457684900358fed9098"
server_key_iv: "395afc2ca9a20f20106c216a36ac1c04"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.298337 18169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:13.300974 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.302229 18169 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:15:13.302340 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "86730a9c0a684f52809759c01515bba2"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b1afe2fc66662457684900358fed9098"
server_key_iv: "395afc2ca9a20f20106c216a36ac1c04"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.302445 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:13.316879 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:13.317632 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:13.317806 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:13.318389 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:13.319337 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:13.319449 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.319509 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:13.319540 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.328373 18169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46061
I20260924 07:15:13.328402 18296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46061 every 8 connection(s)
I20260924 07:15:13.329285 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:13.336319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18169
I20260924 07:15:13.336448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:13.336699 392 external_mini_cluster.cc:1428] Setting key 9b85c8d64c4c0e7d42632a1fa5c7bab2
I20260924 07:15:13.338407 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.331236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33101 (local address 127.0.98.62:45775)
0924 07:15:13.331431 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.331435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.332334 (+ 899us) server_negotiation.cc:407] Connection header received
0924 07:15:13.333653 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.333656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.333705 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.333779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:13.334632 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.335192 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.336422 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.336685 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.336774 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.337914 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:13.337918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:13.337921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:13.337934 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:13.337952 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:13.337970 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:13.338044 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.338208 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.338269 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:15:13.338737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--builtin_ntp_servers=127.0.98.20:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:13.339265 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.331549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.98.2:33101)
0924 07:15:13.332191 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.332232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.333456 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.333929 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.333935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.334023 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:13.334497 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.334510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.335334 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.335337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.336278 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.336287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.336843 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.336847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.337015 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.337642 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:13.337644 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:13.337698 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:13.337762 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:13.338066 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:13.338074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:13.338078 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.338189 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.338472 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":99,"threads_started":1}
I20260924 07:15:13.340783 18297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45775
I20260924 07:15:13.341046 18297 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:13.341661 18297 heartbeater.cc:511] Master 127.0.98.62:45775 requested a full tablet report, sending...
I20260924 07:15:13.342778 17976 ts_manager.cc:194] Registered new tserver with Master: 86730a9c0a684f52809759c01515bba2 (127.0.98.2:46061)
I20260924 07:15:13.343360 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33101
W20260924 07:15:13.452879 18300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:13.453150 18300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:13.453188 18300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:13.457378 18300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:13.457453 18300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:13.457509 18300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:13.462488 18300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:13.463619 18300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:13.464998 18300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:13.474514 18305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.474681 18308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:13.474689 18306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:13.474802 18300 server_base.cc:1034] running on GCE node
I20260924 07:15:13.475318 18300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:13.475996 18300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:13.477177 18300 hybrid_clock.cc:648] HybridClock initialized: now 1790234113477166 us; error 28 us; skew 500 ppm
I20260924 07:15:13.479228 18300 webserver.cc:460] Webserver started at http://127.0.98.3:43853/ using document root <none> and password file <none>
I20260924 07:15:13.479849 18300 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:13.479920 18300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:13.480115 18300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:13.482198 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "02c69b81e64b429498d6c825144d62a6"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b8093da61f283d25cded907aa6847734"
server_key_iv: "f34a4d9d344aa1b1f59aa7b3ae1403d9"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.482766 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02c69b81e64b429498d6c825144d62a6"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b8093da61f283d25cded907aa6847734"
server_key_iv: "f34a4d9d344aa1b1f59aa7b3ae1403d9"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.486685 18300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:13.489295 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.490701 18300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 07:15:13.490839 18300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "02c69b81e64b429498d6c825144d62a6"
format_stamp: "Formatted at 2026-09-24 07:15:13 on dist-test-slave-cnv3"
server_key: "b8093da61f283d25cded907aa6847734"
server_key_iv: "f34a4d9d344aa1b1f59aa7b3ae1403d9"
server_key_version: "encryptionkey@0"
I20260924 07:15:13.490960 18300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:13.503172 18300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:13.503938 18300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:13.504114 18300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:13.504698 18300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:13.505661 18300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:13.505707 18300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.505759 18300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:13.505787 18300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:13.514243 18300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35759
I20260924 07:15:13.514264 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35759 every 8 connection(s)
I20260924 07:15:13.515159 18300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:13.524232 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.517084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46419 (local address 127.0.98.62:45775)
0924 07:15:13.517215 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.517219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.518223 (+ 1004us) server_negotiation.cc:407] Connection header received
0924 07:15:13.519646 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.519649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.519709 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.519785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:13.520677 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.521205 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.522331 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.522546 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.522634 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.523755 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:13.523759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:13.523762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:13.523777 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:13.523797 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:13.523805 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:13.523863 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.524006 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.524079 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:15:13.525111 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.517446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45775 (local address 127.0.98.3:46419)
0924 07:15:13.518079 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.518119 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.519426 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.519962 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.519969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.520056 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:13.520525 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.520536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.521322 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.521325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.522210 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.522219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.522685 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.522689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.522847 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.523484 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:13.523486 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:13.523526 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:13.523604 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:13.523873 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:13.523883 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:13.523889 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.523997 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.524281 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260924 07:15:13.525444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18300
I20260924 07:15:13.525550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:13.525842 392 external_mini_cluster.cc:1428] Setting key 9223178c3502170fe7c7ba508cae5d1e
I20260924 07:15:13.526350 18428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45775
I20260924 07:15:13.526626 18428 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:13.527320 18428 heartbeater.cc:511] Master 127.0.98.62:45775 requested a full tablet report, sending...
I20260924 07:15:13.528452 17976 ts_manager.cc:194] Registered new tserver with Master: 02c69b81e64b429498d6c825144d62a6 (127.0.98.3:35759)
I20260924 07:15:13.529095 17976 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46419
I20260924 07:15:13.540132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:13.545676 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.542052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.98.62:45775)
0924 07:15:13.542203 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.542207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.542313 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:15:13.542420 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.542422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.542464 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.542535 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:13.542761 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.543275 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.544204 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.544429 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.544517 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.545254 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:13.545258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:13.545260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:13.545274 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:13.545293 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:13.545301 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:13.545364 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.545429 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.545548 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:15:13.548666 17976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:13.551986 17976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:13.567335 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.561676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.0.98.2:46061)
0924 07:15:13.562441 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.562452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.562477 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:15:13.562554 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.562559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.562784 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.563140 (+ 356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:13.563253 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.564030 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.565526 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.566383 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.566499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.566708 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.567034 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.567151 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":109,"threads_started":1}
I20260924 07:15:13.567361 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.560907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33465 (local address 127.0.0.1:43256)
0924 07:15:13.561375 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.561387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.561464 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.562789 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.562792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.562810 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:13.562921 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.562928 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.564239 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.564242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.565555 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.565564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.566561 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.566564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.566779 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.566860 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.566976 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.567168 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":181,"thread_start_us":84,"threads_started":1}
I20260924 07:15:13.567395 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.561488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46061 (local address 127.0.0.1:38948)
0924 07:15:13.561697 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.561708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.561785 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.562932 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.562935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.562950 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:13.563019 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.563025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.564153 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.564156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.565389 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.565400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.566528 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.566532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.566780 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.566857 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.566976 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.567169 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":176,"thread_start_us":95,"threads_started":1}
I20260924 07:15:13.568053 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.562024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.0.98.1:33465)
0924 07:15:13.562327 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.562344 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.562368 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:15:13.562440 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.562444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.562644 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.562813 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:13.563011 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.564139 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.565679 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.566413 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.566541 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.566708 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.567768 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.567876 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260924 07:15:13.568076 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.560747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35759 (local address 127.0.0.1:51056)
0924 07:15:13.561272 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.561327 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.561465 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.562253 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.562260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.562285 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:13.562437 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.562445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.564073 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.564077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.565553 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.565564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.567690 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.567695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.567784 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.567808 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.567874 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.567946 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260924 07:15:13.568430 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.560921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51056 (local address 127.0.98.3:35759)
0924 07:15:13.561739 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.561750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.561774 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:15:13.561848 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.561854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.562073 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.562592 (+ 519us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:13.562732 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.563947 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.566777 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.567569 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.567681 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.567862 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.567905 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.567985 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":195,"threads_started":1}
I20260924 07:15:13.571122 18100 tablet_service.cc:1467] Processing CreateTablet for tablet 7bba55d8c472460aba5f04d94da3ad57 (DEFAULT_TABLE table=test-table [id=709549b936e84de2a9ef64222d281934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:13.571054 18231 tablet_service.cc:1467] Processing CreateTablet for tablet 7bba55d8c472460aba5f04d94da3ad57 (DEFAULT_TABLE table=test-table [id=709549b936e84de2a9ef64222d281934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:13.572234 18231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bba55d8c472460aba5f04d94da3ad57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:13.572234 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bba55d8c472460aba5f04d94da3ad57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:13.572161 18362 tablet_service.cc:1467] Processing CreateTablet for tablet 7bba55d8c472460aba5f04d94da3ad57 (DEFAULT_TABLE table=test-table [id=709549b936e84de2a9ef64222d281934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:13.573180 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bba55d8c472460aba5f04d94da3ad57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:13.578693 18445 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Bootstrap starting.
I20260924 07:15:13.580396 18447 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Bootstrap starting.
I20260924 07:15:13.581090 18446 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Bootstrap starting.
I20260924 07:15:13.581199 18445 tablet_bootstrap.cc:654] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:13.582090 18445 log.cc:824] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:13.582594 18447 tablet_bootstrap.cc:654] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:13.582801 18446 tablet_bootstrap.cc:654] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:13.583411 18447 log.cc:824] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:13.583602 18446 log.cc:824] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:13.584941 18445 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: No bootstrap required, opened a new log
I20260924 07:15:13.585170 18445 ts_tablet_manager.cc:1397] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:15:13.585390 18446 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: No bootstrap required, opened a new log
I20260924 07:15:13.585561 18446 ts_tablet_manager.cc:1397] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:15:13.585561 18447 tablet_bootstrap.cc:492] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: No bootstrap required, opened a new log
I20260924 07:15:13.585733 18447 ts_tablet_manager.cc:1397] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:15:13.589003 18446 raft_consensus.cc:348] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.589291 18446 raft_consensus.cc:374] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:13.589339 18446 raft_consensus.cc:729] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86730a9c0a684f52809759c01515bba2, State: Initialized, Role: FOLLOWER
I20260924 07:15:13.589644 18445 raft_consensus.cc:348] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.589805 18446 consensus_queue.cc:260] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.589915 18445 raft_consensus.cc:374] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:13.589785 18447 raft_consensus.cc:348] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.589969 18445 raft_consensus.cc:729] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3125a31f904d4201939d65257979043e, State: Initialized, Role: FOLLOWER
I20260924 07:15:13.589985 18447 raft_consensus.cc:374] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:13.590035 18447 raft_consensus.cc:729] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c69b81e64b429498d6c825144d62a6, State: Initialized, Role: FOLLOWER
I20260924 07:15:13.590441 18445 consensus_queue.cc:260] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.590441 18447 consensus_queue.cc:260] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.591020 18446 ts_tablet_manager.cc:1428] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260924 07:15:13.591429 18447 ts_tablet_manager.cc:1428] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:15:13.591429 18297 heartbeater.cc:503] Master 127.0.98.62:45775 was elected leader, sending a full tablet report...
I20260924 07:15:13.591751 18428 heartbeater.cc:503] Master 127.0.98.62:45775 was elected leader, sending a full tablet report...
I20260924 07:15:13.592481 18445 ts_tablet_manager.cc:1428] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 07:15:13.592834 18166 heartbeater.cc:503] Master 127.0.98.62:45775 was elected leader, sending a full tablet report...
W20260924 07:15:13.637571 18167 tablet.cc:2369] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:13.751008 18452 raft_consensus.cc:482] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:13.751212 18452 raft_consensus.cc:504] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.752346 18452 leader_election.cc:288] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3125a31f904d4201939d65257979043e (127.0.98.1:33465), 86730a9c0a684f52809759c01515bba2 (127.0.98.2:46061)
I20260924 07:15:13.756531 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.752593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33465 (local address 127.0.98.3:43761)
0924 07:15:13.752753 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.752769 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.752855 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.753228 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.753231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.753249 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:13.753360 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.753367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.754272 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.754275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.755175 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.755192 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.756067 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.756071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.756175 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.756230 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.756311 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.756389 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:15:13.756536 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.752779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43761 (local address 127.0.98.1:33465)
0924 07:15:13.752931 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.752936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.752954 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:15:13.753033 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.753036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.753086 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.753174 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:13.753483 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.754158 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.755338 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.755951 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.756033 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.756097 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.756332 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.756405 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:15:13.756963 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.753029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48753 (local address 127.0.98.2:46061)
0924 07:15:13.753152 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.753156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.753344 (+ 188us) server_negotiation.cc:407] Connection header received
0924 07:15:13.753426 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.753429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.753484 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.753539 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:13.753883 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.754438 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.755825 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.756326 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.756417 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.756474 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.756768 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.756855 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:15:13.757298 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.752877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46061 (local address 127.0.98.3:48753)
0924 07:15:13.753242 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.753255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:13.753347 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:13.753636 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:13.753639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:13.753658 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:13.753778 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.753784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.754615 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.754617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.755693 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:13.755704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.756442 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.756445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:13.756541 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.756638 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:15:13.757106 (+ 468us) client_negotiation.cc:240] Negotiation successful
0924 07:15:13.757194 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260924 07:15:13.757277 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bba55d8c472460aba5f04d94da3ad57" candidate_uuid: "02c69b81e64b429498d6c825144d62a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3125a31f904d4201939d65257979043e" is_pre_election: true
I20260924 07:15:13.757588 18120 raft_consensus.cc:2454] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c69b81e64b429498d6c825144d62a6 in term 0.
I20260924 07:15:13.757859 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bba55d8c472460aba5f04d94da3ad57" candidate_uuid: "02c69b81e64b429498d6c825144d62a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86730a9c0a684f52809759c01515bba2" is_pre_election: true
I20260924 07:15:13.758149 18251 raft_consensus.cc:2454] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c69b81e64b429498d6c825144d62a6 in term 0.
I20260924 07:15:13.758126 18316 leader_election.cc:302] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c69b81e64b429498d6c825144d62a6, 3125a31f904d4201939d65257979043e; no voters:
I20260924 07:15:13.758391 18452 raft_consensus.cc:2781] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:13.758482 18452 raft_consensus.cc:482] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:13.758533 18452 raft_consensus.cc:3037] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:13.759608 18452 raft_consensus.cc:504] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.759936 18452 leader_election.cc:288] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [CANDIDATE]: Term 1 election: Requested vote from peers 3125a31f904d4201939d65257979043e (127.0.98.1:33465), 86730a9c0a684f52809759c01515bba2 (127.0.98.2:46061)
I20260924 07:15:13.760294 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bba55d8c472460aba5f04d94da3ad57" candidate_uuid: "02c69b81e64b429498d6c825144d62a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3125a31f904d4201939d65257979043e"
I20260924 07:15:13.760329 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bba55d8c472460aba5f04d94da3ad57" candidate_uuid: "02c69b81e64b429498d6c825144d62a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86730a9c0a684f52809759c01515bba2"
I20260924 07:15:13.760414 18120 raft_consensus.cc:3037] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:13.760448 18251 raft_consensus.cc:3037] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:13.761471 18120 raft_consensus.cc:2454] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c69b81e64b429498d6c825144d62a6 in term 1.
I20260924 07:15:13.761663 18251 raft_consensus.cc:2454] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c69b81e64b429498d6c825144d62a6 in term 1.
I20260924 07:15:13.761787 18316 leader_election.cc:302] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c69b81e64b429498d6c825144d62a6, 3125a31f904d4201939d65257979043e; no voters:
I20260924 07:15:13.762008 18452 raft_consensus.cc:2781] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:13.762223 18452 raft_consensus.cc:686] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 1 LEADER]: Becoming Leader. State: Replica: 02c69b81e64b429498d6c825144d62a6, State: Running, Role: LEADER
I20260924 07:15:13.762543 18452 consensus_queue.cc:237] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } }
I20260924 07:15:13.765796 17976 catalog_manager.cc:5495] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02c69b81e64b429498d6c825144d62a6 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02c69b81e64b429498d6c825144d62a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02c69b81e64b429498d6c825144d62a6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 } health_report { overall_health: UNKNOWN } } }
W20260924 07:15:13.766047 18429 tablet.cc:2369] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:13.803557 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.800042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.98.3:35759)
0924 07:15:13.800166 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:13.800169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:13.800182 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:15:13.800259 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:13.800262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:13.800313 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:13.800398 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:13.800716 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.801238 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.802166 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:13.802393 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:13.802473 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:13.802724 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:13.803147 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:13.803236 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:13.803334 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:15:13.803419 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:15:13.809801 18251 raft_consensus.cc:1261] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 1 FOLLOWER]: Refusing update from remote peer 02c69b81e64b429498d6c825144d62a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:13.809803 18120 raft_consensus.cc:1261] T 7bba55d8c472460aba5f04d94da3ad57 P 3125a31f904d4201939d65257979043e [term 1 FOLLOWER]: Refusing update from remote peer 02c69b81e64b429498d6c825144d62a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:13.810672 18455 consensus_queue.cc:1035] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3125a31f904d4201939d65257979043e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:13.810913 18452 consensus_queue.cc:1035] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86730a9c0a684f52809759c01515bba2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:13.818641 18458 mvcc.cc:205] Tried to move back new op lower bound from 7332798930156560384 to 7332798929974128640. Current Snapshot: MvccSnapshot[applied={T|T < 7332798930156560384}]
I20260924 07:15:13.818915 18460 mvcc.cc:205] Tried to move back new op lower bound from 7332798930156560384 to 7332798929974128640. Current Snapshot: MvccSnapshot[applied={T|T < 7332798930156560384}]
I20260924 07:15:13.826859 17976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:13.827062 17976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:13.829591 17976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4f077802f6a47ddb15ddd550decf7e4: Sending DeleteTablet for 3 replicas of tablet 7bba55d8c472460aba5f04d94da3ad57
I20260924 07:15:13.830547 18362 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bba55d8c472460aba5f04d94da3ad57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:13 UTC) from {username='slave'} at 127.0.0.1:51056
I20260924 07:15:13.830981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18038
I20260924 07:15:13.831146 18231 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bba55d8c472460aba5f04d94da3ad57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:13 UTC) from {username='slave'} at 127.0.0.1:38948
I20260924 07:15:13.831702 18467 tablet_replica.cc:330] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: stopping tablet replica
I20260924 07:15:13.832036 18467 raft_consensus.cc:2229] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:15:13.832384 18467 raft_consensus.cc:2258] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:13.833911 18467 ts_tablet_manager.cc:1905] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:13.833985 18468 tablet_replica.cc:330] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: stopping tablet replica
I20260924 07:15:13.834304 18468 raft_consensus.cc:2229] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:13.834645 18468 raft_consensus.cc:2258] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:13.836184 18468 ts_tablet_manager.cc:1905] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:13.837162 18467 ts_tablet_manager.cc:1918] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:13.837306 18467 log.cc:1196] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/7bba55d8c472460aba5f04d94da3ad57
I20260924 07:15:13.837666 18467 ts_tablet_manager.cc:1939] T 7bba55d8c472460aba5f04d94da3ad57 P 02c69b81e64b429498d6c825144d62a6: Deleting consensus metadata
I20260924 07:15:13.838761 18468 ts_tablet_manager.cc:1918] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:13.838853 18468 log.cc:1196] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/7bba55d8c472460aba5f04d94da3ad57
W20260924 07:15:13.838914 17963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33465 (error 108)
I20260924 07:15:13.839118 18468 ts_tablet_manager.cc:1939] T 7bba55d8c472460aba5f04d94da3ad57 P 86730a9c0a684f52809759c01515bba2: Deleting consensus metadata
I20260924 07:15:13.838924 17965 catalog_manager.cc:4841] TS 02c69b81e64b429498d6c825144d62a6 (127.0.98.3:35759): tablet 7bba55d8c472460aba5f04d94da3ad57 (table test-table [id=709549b936e84de2a9ef64222d281934]) successfully deleted
I20260924 07:15:13.839745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18169
I20260924 07:15:13.840140 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.839830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33465 (local address 127.0.0.1:43266)
0924 07:15:13.839966 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.840048 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:15:13.840385 17963 catalog_manager.cc:4584] TS 3125a31f904d4201939d65257979043e (127.0.98.1:33465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bba55d8c472460aba5f04d94da3ad57: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33465: connect: Connection refused (error 111)
I20260924 07:15:13.845220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18300
I20260924 07:15:13.846422 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:13.846137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46061 (local address 127.0.0.1:38952)
0924 07:15:13.846275 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:13.846331 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 07:15:13.850745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 17947
2026-09-24T07:15:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:13 dist-test-slave-cnv3 krb5kdc[18472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:13 dist-test-slave-cnv3 krb5kdc[18472](info): set up 2 sockets
Sep 24 07:15:13 dist-test-slave-cnv3 krb5kdc[18472](info): commencing operation
krb5kdc: starting...
W20260924 07:15:14.811281 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:14 dist-test-slave-cnv3 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:14.933931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36901
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:15.044646 18485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:15.044979 18485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:15.045038 18485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:15.049230 18485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:15.049309 18485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:15.049336 18485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:15.049356 18485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:15.049376 18485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:15.055630 18485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:15.056993 18485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:15.058203 18485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:15.067289 18493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.067303 18490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.067301 18491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:15.067713 18485 server_base.cc:1034] running on GCE node
I20260924 07:15:15.068310 18485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:15.069571 18485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:15.070779 18485 hybrid_clock.cc:648] HybridClock initialized: now 1790234115070727 us; error 65 us; skew 500 ppm
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:15.079651 18485 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:15.080680 18485 webserver.cc:460] Webserver started at http://127.0.98.62:40431/ using document root <none> and password file <none>
I20260924 07:15:15.081254 18485 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:15.081319 18485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:15.081521 18485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:15.083688 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "65d7e14d28684f9fa2e8e81510548112"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "d5759a94b18ba30864fd6ce0a9b846f1"
server_key_iv: "e6f24fb8d68c38b640e23370df79153a"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.084244 18485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "65d7e14d28684f9fa2e8e81510548112"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "d5759a94b18ba30864fd6ce0a9b846f1"
server_key_iv: "e6f24fb8d68c38b640e23370df79153a"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.088230 18485 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:15:15.090858 18500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.092329 18485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:15:15.092453 18485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "65d7e14d28684f9fa2e8e81510548112"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "d5759a94b18ba30864fd6ce0a9b846f1"
server_key_iv: "e6f24fb8d68c38b640e23370df79153a"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.092576 18485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:15.113689 18485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:15.116782 18485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:15.116954 18485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:15.123827 18485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36901
I20260924 07:15:15.123924 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36901 every 8 connection(s)
I20260924 07:15:15.124778 18485 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:15.127585 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:15.130604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18485
I20260924 07:15:15.130751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:15.130988 392 external_mini_cluster.cc:1428] Setting key ff5fb0be9ba189224ed746ca83926cdb
I20260924 07:15:15.133476 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: Bootstrap starting.
I20260924 07:15:15.135694 18553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:15.136423 18553 log.cc:824] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:15.138269 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: No bootstrap required, opened a new log
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:15.141758 18553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } }
I20260924 07:15:15.141990 18553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:15.142045 18553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d7e14d28684f9fa2e8e81510548112, State: Initialized, Role: FOLLOWER
I20260924 07:15:15.142515 18553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } }
I20260924 07:15:15.142652 18553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:15.142717 18553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:15.142812 18553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:15.143879 18553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } }
I20260924 07:15:15.144212 18553 leader_election.cc:302] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d7e14d28684f9fa2e8e81510548112; no voters:
I20260924 07:15:15.144452 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.132385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.0.98.62:36901)
0924 07:15:15.132827 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.132843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.132879 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:15.133533 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.133553 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.133907 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.134218 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:15.134986 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.136052 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.136913 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.137379 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.137529 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.139847 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:15.139853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:15.139866 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:15.139893 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:15.141481 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.141907 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.141912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.141916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.142025 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.142375 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.142379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.142381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.142816 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:15.142994 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.143297 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.143610 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260924 07:15:15.144500 18553 leader_election.cc:288] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:15.144616 18558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:15.144830 18558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [term 1 LEADER]: Becoming Leader. State: Replica: 65d7e14d28684f9fa2e8e81510548112, State: Running, Role: LEADER
I20260924 07:15:15.145129 18558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } }
I20260924 07:15:15.145701 18553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:15.147720 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d7e14d28684f9fa2e8e81510548112. Latest consensus state: current_term: 1 leader_uuid: "65d7e14d28684f9fa2e8e81510548112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } }
I20260924 07:15:15.147826 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:15.147804 18559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d7e14d28684f9fa2e8e81510548112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d7e14d28684f9fa2e8e81510548112" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36901 } } }
I20260924 07:15:15.147907 18559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:15.148259 18573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:15.151278 18573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:15.156991 18573 catalog_manager.cc:1347] Generated new cluster ID: c0b85d88e47a4b4b9a07d22ea9ad8605
I20260924 07:15:15.157075 18573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:15.173946 18573 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:15.175254 18573 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:15.186280 18573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: Generated new TSK 0
I20260924 07:15:15.187155 18573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:15:15.219624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:15.333038 18579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:15.333325 18579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:15.333369 18579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:15.337648 18579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:15.337724 18579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:15.337786 18579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:15.343322 18579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:15.344489 18579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:15.345563 18579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:15.355602 18587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.355651 18585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.355720 18584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:15.355890 18579 server_base.cc:1034] running on GCE node
I20260924 07:15:15.356310 18579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:15.356971 18579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:15.358152 18579 hybrid_clock.cc:648] HybridClock initialized: now 1790234115358123 us; error 41 us; skew 500 ppm
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:15.361070 18579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:15:15.361990 18579 webserver.cc:460] Webserver started at http://127.0.98.1:40585/ using document root <none> and password file <none>
I20260924 07:15:15.362603 18579 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:15.362676 18579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:15.362886 18579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:15.365164 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "757836d23aaf4ac5baf68760ddc09b88"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "57b2b07e2afe1227c7d75b63fdc58c78"
server_key_iv: "b2569a4dcec536eb158b019ce408b5b0"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.365741 18579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "757836d23aaf4ac5baf68760ddc09b88"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "57b2b07e2afe1227c7d75b63fdc58c78"
server_key_iv: "b2569a4dcec536eb158b019ce408b5b0"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.369793 18579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:15.372655 18594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.374089 18579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:15.374219 18579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "757836d23aaf4ac5baf68760ddc09b88"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "57b2b07e2afe1227c7d75b63fdc58c78"
server_key_iv: "b2569a4dcec536eb158b019ce408b5b0"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.374338 18579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:15.396673 18579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:15.400027 18579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:15.400214 18579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:15.400789 18579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:15.401633 18579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:15.401680 18579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.401732 18579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:15.401758 18579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.412177 18579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40949
I20260924 07:15:15.413219 18579 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:15.414047 18707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40949 every 8 connection(s)
I20260924 07:15:15.416198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18579
I20260924 07:15:15.416330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:15.416671 392 external_mini_cluster.cc:1428] Setting key 7d989a5400d4380dedfd7149d7efa652
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:15:15.427574 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.415204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36827 (local address 127.0.98.62:36901)
0924 07:15:15.415342 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.415346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.416363 (+ 1017us) server_negotiation.cc:407] Connection header received
0924 07:15:15.417892 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.417895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.417943 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.418025 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:15.419422 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.419962 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.421102 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.421330 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.421418 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.424049 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:15.424052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:15.424054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:15.424080 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:15.425453 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.426000 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.426005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.426006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.426097 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.426453 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.426456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.426457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.426651 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:15.426730 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.427284 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.427439 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:15:15.428465 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.415643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.98.1:36827)
0924 07:15:15.416183 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.416245 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.417664 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.418379 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.418387 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.418747 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:15.419248 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.419261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.420086 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.420089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.420987 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.420996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.421484 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.421488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.421641 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.422195 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:15.422212 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:15.423868 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:15.425564 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.425568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.425582 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.425844 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.426184 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.426186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.426187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.426332 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.426729 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:15.426735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:15.427002 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.427189 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.427524 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":113,"threads_started":1}
I20260924 07:15:15.429800 18708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36901
I20260924 07:15:15.430043 18708 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:15.430549 18708 heartbeater.cc:511] Master 127.0.98.62:36901 requested a full tablet report, sending...
I20260924 07:15:15.432132 18517 ts_manager.cc:194] Registered new tserver with Master: 757836d23aaf4ac5baf68760ddc09b88 (127.0.98.1:40949)
I20260924 07:15:15.433269 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:15:15.447945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:15.561596 18713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:15.561888 18713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:15.561933 18713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:15.566212 18713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:15.566290 18713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:15.566358 18713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:15.572016 18713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:15.573117 18713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:15.574177 18713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:15.583863 18718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.583863 18719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.583877 18721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:15.584136 18713 server_base.cc:1034] running on GCE node
I20260924 07:15:15.584717 18713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:15.585518 18713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:15.586719 18713 hybrid_clock.cc:648] HybridClock initialized: now 1790234115586699 us; error 32 us; skew 500 ppm
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:15.589578 18713 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:15:15.590479 18713 webserver.cc:460] Webserver started at http://127.0.98.2:38373/ using document root <none> and password file <none>
I20260924 07:15:15.591054 18713 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:15.591122 18713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:15.591310 18713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:15.593456 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "acb2ff282c724fe9baddefb97cf4173f"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "f13db9b3f78322d4132d3991efcbeb5c"
server_key_iv: "12bad0a93471642c815c946c7e97084e"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.594034 18713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acb2ff282c724fe9baddefb97cf4173f"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "f13db9b3f78322d4132d3991efcbeb5c"
server_key_iv: "12bad0a93471642c815c946c7e97084e"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.598026 18713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 07:15:15.600584 18728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.601818 18713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:15.601943 18713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "acb2ff282c724fe9baddefb97cf4173f"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "f13db9b3f78322d4132d3991efcbeb5c"
server_key_iv: "12bad0a93471642c815c946c7e97084e"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.602046 18713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:15.612908 18713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:15.616024 18713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:15.616199 18713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:15.616770 18713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:15.617580 18713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:15.617626 18713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.617679 18713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:15.617705 18713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.625967 18713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39643
I20260924 07:15:15.625993 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39643 every 8 connection(s)
I20260924 07:15:15.626891 18713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:15.634756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18713
I20260924 07:15:15.634963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:15.635283 392 external_mini_cluster.cc:1428] Setting key db179399dda908fe390713bbc5e1c176
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:15.641155 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.628906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39009 (local address 127.0.98.62:36901)
0924 07:15:15.629065 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.629068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.629871 (+ 803us) server_negotiation.cc:407] Connection header received
0924 07:15:15.631211 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.631215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.631300 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.631402 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:15.632642 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.633172 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.634319 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.634535 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.634624 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.637205 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:15.637208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:15.637210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:15.637234 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:15.638826 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.639346 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.639350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.639352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.639471 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.639934 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.639937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.639938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.640136 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:15.640221 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.640922 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.641035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:15:15.642293 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.629254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.98.2:39009)
0924 07:15:15.629697 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.629744 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.630984 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.631597 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.631604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.631995 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:15.632492 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.632506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.633293 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.633296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.634191 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.634199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.634679 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.634683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.634855 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.635515 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:15.635531 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:15.637026 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:15.638973 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.638978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.638991 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.639252 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.639555 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.639557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.639559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.639732 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.640244 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:15.640251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:15.640589 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.640807 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.641187 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260924 07:15:15.643703 18842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36901
I20260924 07:15:15.643975 18842 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:15.644594 18842 heartbeater.cc:511] Master 127.0.98.62:36901 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:15:15.645833 18517 ts_manager.cc:194] Registered new tserver with Master: acb2ff282c724fe9baddefb97cf4173f (127.0.98.2:39643)
I20260924 07:15:15.646450 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:15:15.662814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:15.774461 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:15.774765 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:15.774830 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:15.779064 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:15.779139 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:15.779219 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:15.784858 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:15.785979 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:15.787005 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:15.796730 18853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.796737 18852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:15.796836 18855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:15.796967 18847 server_base.cc:1034] running on GCE node
I20260924 07:15:15.797505 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:15.798171 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:15.799336 18847 hybrid_clock.cc:648] HybridClock initialized: now 1790234115799284 us; error 69 us; skew 500 ppm
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:15.802239 18847 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:15:15.803175 18847 webserver.cc:460] Webserver started at http://127.0.98.3:32879/ using document root <none> and password file <none>
I20260924 07:15:15.803777 18847 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:15.803838 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:15.804032 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:15.806241 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "b07e6f20ae304c4aaea66b55dd8f8d70"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "3a938c7fd5736c824859efded2d299ef"
server_key_iv: "d0a111fcac68cb63818a5152a1bc2771"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.806778 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b07e6f20ae304c4aaea66b55dd8f8d70"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "3a938c7fd5736c824859efded2d299ef"
server_key_iv: "d0a111fcac68cb63818a5152a1bc2771"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.810573 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:15.812975 18862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.814189 18847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:15.814332 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "b07e6f20ae304c4aaea66b55dd8f8d70"
format_stamp: "Formatted at 2026-09-24 07:15:15 on dist-test-slave-cnv3"
server_key: "3a938c7fd5736c824859efded2d299ef"
server_key_iv: "d0a111fcac68cb63818a5152a1bc2771"
server_key_version: "encryptionkey@0"
I20260924 07:15:15.814435 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:15.831729 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:15.834679 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:15.834862 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:15.835486 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:15.836308 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:15.836357 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.836419 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:15.836448 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:15.845618 18847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37121
I20260924 07:15:15.845660 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37121 every 8 connection(s)
I20260924 07:15:15.846647 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:15.849480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 18847
I20260924 07:15:15.849618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:15.849932 392 external_mini_cluster.cc:1428] Setting key 10b9a655ff5946a86273c5f4f8f8b3c5
Sep 24 07:15:15 dist-test-slave-cnv3 krb5kdc[18472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:15.860530 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.848626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33375 (local address 127.0.98.62:36901)
0924 07:15:15.848765 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.848768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.849688 (+ 920us) server_negotiation.cc:407] Connection header received
0924 07:15:15.851218 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.851222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.851282 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.851359 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:15.852680 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.853249 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.854429 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.854658 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.854738 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.857204 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:15.857207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:15.857209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:15.857236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:15.858584 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.859087 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.859089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.859090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.859163 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.859481 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.859485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.859486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.859691 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:15.859776 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.860289 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.860405 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:15:15.861390 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.848982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36901 (local address 127.0.98.3:33375)
0924 07:15:15.849520 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.849582 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.850995 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.851591 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.851598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.851974 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:15.852548 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.852563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.853369 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.853372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.854310 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.854319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.854799 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.854803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.854970 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.855578 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:15.855594 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:15.857038 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:15.858689 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.858694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.858709 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.858961 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.859231 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:15.859233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:15.859235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:15.859369 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:15.859792 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:15.859797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:15.860056 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.860213 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.860514 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
I20260924 07:15:15.862718 18976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36901
I20260924 07:15:15.862993 18976 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:15.863632 18976 heartbeater.cc:511] Master 127.0.98.62:36901 requested a full tablet report, sending...
I20260924 07:15:15.864684 18517 ts_manager.cc:194] Registered new tserver with Master: b07e6f20ae304c4aaea66b55dd8f8d70 (127.0.98.3:37121)
I20260924 07:15:15.865247 18517 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33375
I20260924 07:15:15.876437 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:15.886268 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.878510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.98.62:36901)
0924 07:15:15.878632 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.878636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.878781 (+ 145us) server_negotiation.cc:407] Connection header received
0924 07:15:15.878862 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.878865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.878917 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.878991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:15.879721 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.880259 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.881114 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.881369 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.881462 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.883048 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:15.883051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:15.883053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:15.883079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:15.884469 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.885028 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.885030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.885031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.885106 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:15.885407 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:15.885409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:15.885410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:15.885609 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:15.885718 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.886024 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.886105 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:15:15.889390 18517 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:15.892526 18517 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:15.907127 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.901999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.0.1:54340)
0924 07:15:15.902301 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.902313 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.902384 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.903025 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.903032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.903055 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:15.903217 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.903224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.904273 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.904276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.905486 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.905496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.906579 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.906582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.906689 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.906723 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.906827 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.906963 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":16,"spinlock_wait_cycles":2432,"thread_start_us":90,"threads_started":1}
I20260924 07:15:15.907333 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.902159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54340 (local address 127.0.98.1:40949)
0924 07:15:15.902616 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.902634 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.902654 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:15:15.902715 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.902719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.902869 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.903189 (+ 320us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:15.903332 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.904144 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.905620 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.906469 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.906568 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.906719 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.907050 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.907161 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":83,"threads_started":1}
I20260924 07:15:15.907836 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.901396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37121 (local address 127.0.0.1:40284)
0924 07:15:15.902310 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.902324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.902489 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.903149 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.903152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.903170 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:15.903263 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.903275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.904776 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.904780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.905929 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.905938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.907453 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.907457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.907542 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.907566 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.907640 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.907705 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260924 07:15:15.907932 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.901734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.3:37121)
0924 07:15:15.902155 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.902166 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.902411 (+ 245us) server_negotiation.cc:407] Connection header received
0924 07:15:15.902639 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.902644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.902837 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.902963 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:15.903411 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.904632 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.906064 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.907157 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.907256 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.907413 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.907689 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.907788 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":151,"threads_started":1}
I20260924 07:15:15.908957 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.901345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.98.2:39643)
0924 07:15:15.901675 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:15.901686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:15.902436 (+ 750us) server_negotiation.cc:407] Connection header received
0924 07:15:15.902518 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:15.902523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:15.902729 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:15.902867 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:15.903469 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.904637 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.906958 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.908381 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.908488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.908617 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:15.908729 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 07:15:15.908817 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":123,"threads_started":1}
I20260924 07:15:15.909622 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:15.901181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39643 (local address 127.0.0.1:38246)
0924 07:15:15.902112 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:15.902164 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:15.902302 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:15.903044 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:15.903046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:15.903068 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:15.903339 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.903345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.905626 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.905629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.906836 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:15.906848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:15.908506 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:15.908510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:15.908599 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:15.908622 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:15:15.908702 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:15:15.909493 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":7,"spinlock_wait_cycles":1280,"thread_start_us":142,"threads_started":1}
I20260924 07:15:15.910612 18624 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 (DEFAULT_TABLE table=test-table [id=4a3f4b2d4bb24f0e92277cd1e636741d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:15.910617 18910 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 (DEFAULT_TABLE table=test-table [id=4a3f4b2d4bb24f0e92277cd1e636741d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:15.911650 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5fba4431a7489ca08a2733d876f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:15.911661 18910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5fba4431a7489ca08a2733d876f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:15.912501 18776 tablet_service.cc:1467] Processing CreateTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 (DEFAULT_TABLE table=test-table [id=4a3f4b2d4bb24f0e92277cd1e636741d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:15.913573 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d5fba4431a7489ca08a2733d876f1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:15.917912 18993 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: Bootstrap starting.
I20260924 07:15:15.919606 18994 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Bootstrap starting.
I20260924 07:15:15.920470 18995 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: Bootstrap starting.
I20260924 07:15:15.921211 18993 tablet_bootstrap.cc:654] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:15.921595 18994 tablet_bootstrap.cc:654] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:15.922448 18993 log.cc:824] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:15.922500 18994 log.cc:824] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:15.922955 18995 tablet_bootstrap.cc:654] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:15.923730 18995 log.cc:824] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:15.924542 18993 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: No bootstrap required, opened a new log
I20260924 07:15:15.924556 18994 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: No bootstrap required, opened a new log
I20260924 07:15:15.924746 18994 ts_tablet_manager.cc:1397] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:15:15.924780 18993 ts_tablet_manager.cc:1397] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:15:15.925678 18995 tablet_bootstrap.cc:492] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: No bootstrap required, opened a new log
I20260924 07:15:15.925858 18995 ts_tablet_manager.cc:1397] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:15:15.927672 18993 raft_consensus.cc:348] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.927879 18993 raft_consensus.cc:374] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:15.927911 18993 raft_consensus.cc:729] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757836d23aaf4ac5baf68760ddc09b88, State: Initialized, Role: FOLLOWER
I20260924 07:15:15.928319 18993 consensus_queue.cc:260] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.928524 18994 raft_consensus.cc:348] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.928803 18994 raft_consensus.cc:374] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:15.928879 18994 raft_consensus.cc:729] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07e6f20ae304c4aaea66b55dd8f8d70, State: Initialized, Role: FOLLOWER
I20260924 07:15:15.929353 18708 heartbeater.cc:503] Master 127.0.98.62:36901 was elected leader, sending a full tablet report...
I20260924 07:15:15.929323 18994 consensus_queue.cc:260] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.929519 18995 raft_consensus.cc:348] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.929765 18995 raft_consensus.cc:374] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:15.929816 18995 raft_consensus.cc:729] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb2ff282c724fe9baddefb97cf4173f, State: Initialized, Role: FOLLOWER
I20260924 07:15:15.929859 18993 ts_tablet_manager.cc:1428] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:15:15.930229 18994 ts_tablet_manager.cc:1428] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:15:15.930253 18995 consensus_queue.cc:260] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:15.930562 18976 heartbeater.cc:503] Master 127.0.98.62:36901 was elected leader, sending a full tablet report...
I20260924 07:15:15.931017 18842 heartbeater.cc:503] Master 127.0.98.62:36901 was elected leader, sending a full tablet report...
I20260924 07:15:15.931326 18995 ts_tablet_manager.cc:1428] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 07:15:16.042790 19001 raft_consensus.cc:482] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:16.042999 19001 raft_consensus.cc:504] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:16.044216 19001 leader_election.cc:288] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757836d23aaf4ac5baf68760ddc09b88 (127.0.98.1:40949), b07e6f20ae304c4aaea66b55dd8f8d70 (127.0.98.3:37121)
I20260924 07:15:16.049320 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.044638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58849 (local address 127.0.98.3:37121)
0924 07:15:16.044768 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.044773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.044921 (+ 148us) server_negotiation.cc:407] Connection header received
0924 07:15:16.045021 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.045024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.045067 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.045134 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:16.045509 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.046309 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048070 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.048686 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048788 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.048854 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.049110 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.049186 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:15:16.049319 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.044577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48391 (local address 127.0.98.1:40949)
0924 07:15:16.044685 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.044690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.044795 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:15:16.044966 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.044969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.045025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.045109 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:16.045509 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.046309 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048070 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.048674 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.048853 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.049124 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.049192 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:15:16.049348 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.044478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.98.2:48391)
0924 07:15:16.044668 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.044683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:16.044790 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:16.045131 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:16.045134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:16.045150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:16.045384 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.045391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.046433 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.046437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.047936 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.047948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.048829 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.049027 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.049044 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:15:16.049105 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:15:16.049188 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":225}
I20260924 07:15:16.049355 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.044510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37121 (local address 127.0.98.2:58849)
0924 07:15:16.044837 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.044849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:16.044933 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:16.045143 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:16.045145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:16.045156 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:16.045387 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.045392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.046440 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.046443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.047936 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.047948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.048829 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.048833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.048973 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.049002 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:15:16.049093 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:15:16.049194 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":163,"thread_start_us":114,"threads_started":1}
I20260924 07:15:16.050122 18656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5fba4431a7489ca08a2733d876f1e0" candidate_uuid: "acb2ff282c724fe9baddefb97cf4173f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757836d23aaf4ac5baf68760ddc09b88" is_pre_election: true
I20260924 07:15:16.050119 18930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5fba4431a7489ca08a2733d876f1e0" candidate_uuid: "acb2ff282c724fe9baddefb97cf4173f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" is_pre_election: true
I20260924 07:15:16.050400 18656 raft_consensus.cc:2454] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb2ff282c724fe9baddefb97cf4173f in term 0.
I20260924 07:15:16.050403 18930 raft_consensus.cc:2454] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate acb2ff282c724fe9baddefb97cf4173f in term 0.
I20260924 07:15:16.050943 18732 leader_election.cc:302] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 757836d23aaf4ac5baf68760ddc09b88, acb2ff282c724fe9baddefb97cf4173f, b07e6f20ae304c4aaea66b55dd8f8d70; no voters:
I20260924 07:15:16.051280 19001 raft_consensus.cc:2781] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:16.051403 19001 raft_consensus.cc:482] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:16.051461 19001 raft_consensus.cc:3037] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:16.052531 19001 raft_consensus.cc:504] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:16.052963 19001 leader_election.cc:288] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [CANDIDATE]: Term 1 election: Requested vote from peers 757836d23aaf4ac5baf68760ddc09b88 (127.0.98.1:40949), b07e6f20ae304c4aaea66b55dd8f8d70 (127.0.98.3:37121)
I20260924 07:15:16.053357 18656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5fba4431a7489ca08a2733d876f1e0" candidate_uuid: "acb2ff282c724fe9baddefb97cf4173f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757836d23aaf4ac5baf68760ddc09b88"
I20260924 07:15:16.053491 18930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d5fba4431a7489ca08a2733d876f1e0" candidate_uuid: "acb2ff282c724fe9baddefb97cf4173f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70"
I20260924 07:15:16.053488 18656 raft_consensus.cc:3037] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:16.053618 18930 raft_consensus.cc:3037] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:16.054551 18656 raft_consensus.cc:2454] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb2ff282c724fe9baddefb97cf4173f in term 1.
I20260924 07:15:16.054606 18930 raft_consensus.cc:2454] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate acb2ff282c724fe9baddefb97cf4173f in term 1.
I20260924 07:15:16.054867 18730 leader_election.cc:302] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757836d23aaf4ac5baf68760ddc09b88, acb2ff282c724fe9baddefb97cf4173f; no voters:
I20260924 07:15:16.055042 19001 raft_consensus.cc:2781] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:16.055279 19001 raft_consensus.cc:686] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [term 1 LEADER]: Becoming Leader. State: Replica: acb2ff282c724fe9baddefb97cf4173f, State: Running, Role: LEADER
I20260924 07:15:16.055627 19001 consensus_queue.cc:237] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } }
I20260924 07:15:16.058751 18516 catalog_manager.cc:5495] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f reported cstate change: term changed from 0 to 1, leader changed from <none> to acb2ff282c724fe9baddefb97cf4173f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "acb2ff282c724fe9baddefb97cf4173f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "acb2ff282c724fe9baddefb97cf4173f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 } health_report { overall_health: UNKNOWN } } }
W20260924 07:15:16.097678 18977 tablet.cc:2369] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:16.102824 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.099430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.98.2:39643)
0924 07:15:16.099555 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.099558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.099572 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:15:16.099640 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.099642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.099723 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.099812 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:16.100060 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.100618 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.101464 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.101676 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.101752 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.102000 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:16.102411 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:16.102525 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.102667 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.102704 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:15:16.108749 18930 raft_consensus.cc:1261] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 1 FOLLOWER]: Refusing update from remote peer acb2ff282c724fe9baddefb97cf4173f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:16.108750 18656 raft_consensus.cc:1261] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 1 FOLLOWER]: Refusing update from remote peer acb2ff282c724fe9baddefb97cf4173f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:16.109421 19003 consensus_queue.cc:1035] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [LEADER]: Connected to new peer: Peer: permanent_uuid: "757836d23aaf4ac5baf68760ddc09b88" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:16.109604 19001 consensus_queue.cc:1035] T 0d5fba4431a7489ca08a2733d876f1e0 P acb2ff282c724fe9baddefb97cf4173f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07e6f20ae304c4aaea66b55dd8f8d70" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:16.121019 19008 mvcc.cc:205] Tried to move back new op lower bound from 7332798939573665792 to 7332798939366100992. Current Snapshot: MvccSnapshot[applied={T|T < 7332798939573665792 or (T in {7332798939573665792})}]
I20260924 07:15:16.125604 18514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:16.125756 18514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:16.128012 18514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65d7e14d28684f9fa2e8e81510548112: Sending DeleteTablet for 3 replicas of tablet 0d5fba4431a7489ca08a2733d876f1e0
I20260924 07:15:16.128846 18776 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38246
I20260924 07:15:16.129258 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54340
I20260924 07:15:16.129737 19017 tablet_replica.cc:330] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88: stopping tablet replica
I20260924 07:15:16.130119 19017 raft_consensus.cc:2229] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:16.130141 18910 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d5fba4431a7489ca08a2733d876f1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40284
I20260924 07:15:16.130517 19017 raft_consensus.cc:2258] T 0d5fba4431a7489ca08a2733d876f1e0 P 757836d23aaf4ac5baf68760ddc09b88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:16.131605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18579
W20260924 07:15:16.138442 18502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40949 (error 108)
I20260924 07:15:16.138952 19018 tablet_replica.cc:330] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: stopping tablet replica
I20260924 07:15:16.139197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18713
I20260924 07:15:16.139403 19018 raft_consensus.cc:2229] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:16.139737 19018 raft_consensus.cc:2258] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:16.141100 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.140774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40949 (local address 127.0.0.1:54356)
0924 07:15:16.140953 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.141019 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 07:15:16.141372 18502 catalog_manager.cc:4584] TS 757836d23aaf4ac5baf68760ddc09b88 (127.0.98.1:40949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d5fba4431a7489ca08a2733d876f1e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40949: connect: Connection refused (error 111)
I20260924 07:15:16.141911 19018 ts_tablet_manager.cc:1905] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:16.144670 19018 ts_tablet_manager.cc:1918] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:16.144778 19018 log.cc:1196] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/0d5fba4431a7489ca08a2733d876f1e0
I20260924 07:15:16.145074 19018 ts_tablet_manager.cc:1939] T 0d5fba4431a7489ca08a2733d876f1e0 P b07e6f20ae304c4aaea66b55dd8f8d70: Deleting consensus metadata
I20260924 07:15:16.145622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18847
I20260924 07:15:16.146557 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.145980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39643 (local address 127.0.0.1:38252)
0924 07:15:16.146417 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.146472 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260924 07:15:16.150581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 18485
2026-09-24T07:15:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2300 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T07:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:16Z Disabled control of system clock
I20260924 07:15:16.178884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45635
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:16.289249 19023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:16.289536 19023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:16.289572 19023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:16.293727 19023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:16.293803 19023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:16.293818 19023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:16.293835 19023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:16.293848 19023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:16.298748 19023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:16.299865 19023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:16.301184 19023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:16.309463 19031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.309630 19023 server_base.cc:1034] running on GCE node
W20260924 07:15:16.309526 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:16.309463 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.310321 19023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:16.311532 19023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:16.312692 19023 hybrid_clock.cc:648] HybridClock initialized: now 1790234116312680 us; error 28 us; skew 500 ppm
I20260924 07:15:16.314710 19023 webserver.cc:460] Webserver started at http://127.0.98.62:34039/ using document root <none> and password file <none>
I20260924 07:15:16.315276 19023 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:16.315338 19023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:16.315618 19023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:16.317670 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "8f92df21d61d48a6b1e4be14e79c7a5f"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0d152f5917386af62ba6d63fb1069c06"
server_key_iv: "684445c11eac1c2bfb95c51570e7f67a"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.318210 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f92df21d61d48a6b1e4be14e79c7a5f"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0d152f5917386af62ba6d63fb1069c06"
server_key_iv: "684445c11eac1c2bfb95c51570e7f67a"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.321955 19023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:16.324407 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.325650 19023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:16.325753 19023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "8f92df21d61d48a6b1e4be14e79c7a5f"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0d152f5917386af62ba6d63fb1069c06"
server_key_iv: "684445c11eac1c2bfb95c51570e7f67a"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.325863 19023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:16.340718 19023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:16.341415 19023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:16.341609 19023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:16.348433 19023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45635
I20260924 07:15:16.348484 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45635 every 8 connection(s)
I20260924 07:15:16.349403 19023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:16.352502 19090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:16.354799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19023
I20260924 07:15:16.354899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:16.355151 392 external_mini_cluster.cc:1428] Setting key 273f05733d1240dc018cfc159b2cb62c
I20260924 07:15:16.358743 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: Bootstrap starting.
I20260924 07:15:16.361359 19090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:16.362174 19090 log.cc:824] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:16.364574 19090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: No bootstrap required, opened a new log
I20260924 07:15:16.365629 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.356587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.0.98.62:45635)
0924 07:15:16.357276 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.357296 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.357339 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:15:16.358159 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.358183 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.358674 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.359040 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:16.359519 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.360928 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.361794 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.362499 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.362651 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.363588 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:16.363594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:16.363611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:16.364104 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:16.364173 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:16.364199 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:16.364294 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.364426 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.364700 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260924 07:15:16.368789 19090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260924 07:15:16.369032 19090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:16.369066 19090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f92df21d61d48a6b1e4be14e79c7a5f, State: Initialized, Role: FOLLOWER
I20260924 07:15:16.369437 19090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260924 07:15:16.369554 19090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:16.369592 19090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:16.369666 19090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:16.370635 19090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260924 07:15:16.370970 19090 leader_election.cc:302] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f92df21d61d48a6b1e4be14e79c7a5f; no voters:
I20260924 07:15:16.371256 19090 leader_election.cc:288] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:16.371439 19095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:16.371652 19095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [term 1 LEADER]: Becoming Leader. State: Replica: 8f92df21d61d48a6b1e4be14e79c7a5f, State: Running, Role: LEADER
I20260924 07:15:16.372018 19095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } }
I20260924 07:15:16.372501 19090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:16.373550 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } } }
I20260924 07:15:16.373677 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:16.374066 19097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f92df21d61d48a6b1e4be14e79c7a5f. Latest consensus state: current_term: 1 leader_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f92df21d61d48a6b1e4be14e79c7a5f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45635 } } }
I20260924 07:15:16.374226 19097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:16.377271 19110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:16.377343 19110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:16.377418 19104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:16.378111 19104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:16.384066 19104 catalog_manager.cc:1347] Generated new cluster ID: 6777c24658024830ac81a2cc34ffcccc
I20260924 07:15:16.384153 19104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:16.407863 19104 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:16.409231 19104 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:16.417198 19104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: Generated new TSK 0
I20260924 07:15:16.418064 19104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:16.424703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--builtin_ntp_servers=127.0.98.20:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:16.537226 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:16.537510 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:16.537549 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:16.541662 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:16.541733 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:16.541788 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:16.546640 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:16.547812 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:16.549197 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:16.559605 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:16.559664 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.559778 19114 server_base.cc:1034] running on GCE node
W20260924 07:15:16.559664 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.560361 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:16.561018 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:16.562187 19114 hybrid_clock.cc:648] HybridClock initialized: now 1790234116562164 us; error 34 us; skew 500 ppm
I20260924 07:15:16.564072 19114 webserver.cc:460] Webserver started at http://127.0.98.1:35501/ using document root <none> and password file <none>
I20260924 07:15:16.564640 19114 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:16.564706 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:16.564891 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:16.567044 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "76a85fde65964a38bffd4de8e7beb9e8"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "f03ee7ed09ec50bd46115e73a9752331"
server_key_iv: "250ba809b903e9ea34f90f1ffd711d39"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.567622 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "76a85fde65964a38bffd4de8e7beb9e8"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "f03ee7ed09ec50bd46115e73a9752331"
server_key_iv: "250ba809b903e9ea34f90f1ffd711d39"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.571533 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:15:16.574290 19128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.575702 19114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:16.575821 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "76a85fde65964a38bffd4de8e7beb9e8"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "f03ee7ed09ec50bd46115e73a9752331"
server_key_iv: "250ba809b903e9ea34f90f1ffd711d39"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.575937 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:16.588032 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:16.588737 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:16.588919 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:16.589497 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:16.590437 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:16.590494 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.590548 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:16.590572 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.599443 19114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42569
I20260924 07:15:16.599488 19241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42569 every 8 connection(s)
I20260924 07:15:16.600353 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:16.601183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19114
I20260924 07:15:16.601303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:16.601550 392 external_mini_cluster.cc:1428] Setting key da14cdc723c67a976c3b7459835f091b
I20260924 07:15:16.603819 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--builtin_ntp_servers=127.0.98.20:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:16.609913 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.602201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55817 (local address 127.0.98.62:45635)
0924 07:15:16.602325 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.602330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.603851 (+ 1521us) server_negotiation.cc:407] Connection header received
0924 07:15:16.605264 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.605267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.605324 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.605397 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:16.606315 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.606873 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.608064 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.608278 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.608399 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.609442 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:16.609446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:16.609449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:16.609462 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:16.609481 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:16.609489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:16.609548 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.609687 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.609753 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:15:16.610775 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.602723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45635 (local address 127.0.98.1:55817)
0924 07:15:16.603710 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.603752 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:16.605044 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:16.605587 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:16.605593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:16.605678 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:16.606167 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.606180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.607004 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.607007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.607941 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.607951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.608449 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.608453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.608609 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.609185 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:16.609187 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:16.609228 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:16.609293 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:16.609558 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:16.609567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:16.609573 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:15:16.609680 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:15:16.609967 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":126,"threads_started":1}
I20260924 07:15:16.612150 19242 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45635
I20260924 07:15:16.612386 19242 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:16.612948 19242 heartbeater.cc:511] Master 127.0.98.62:45635 requested a full tablet report, sending...
I20260924 07:15:16.614521 19054 ts_manager.cc:194] Registered new tserver with Master: 76a85fde65964a38bffd4de8e7beb9e8 (127.0.98.1:42569)
I20260924 07:15:16.615763 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55817
W20260924 07:15:16.714836 19245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:16.715135 19245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:16.715229 19245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:16.719513 19245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:16.719576 19245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:16.719655 19245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:16.724661 19245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:16.725801 19245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:16.727200 19245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:16.737012 19253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:16.737000 19250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.737202 19245 server_base.cc:1034] running on GCE node
W20260924 07:15:16.736991 19251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.737721 19245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:16.738420 19245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:16.739603 19245 hybrid_clock.cc:648] HybridClock initialized: now 1790234116739575 us; error 40 us; skew 500 ppm
I20260924 07:15:16.741585 19245 webserver.cc:460] Webserver started at http://127.0.98.2:44289/ using document root <none> and password file <none>
I20260924 07:15:16.742157 19245 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:16.742219 19245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:16.742408 19245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:16.744581 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "de3c4bd1d90b41449fb456652db19afa"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0a103ce02e7b4a8d932522629c7325c3"
server_key_iv: "b5e336276e3506268c41a187d7e1157f"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.745151 19245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "de3c4bd1d90b41449fb456652db19afa"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0a103ce02e7b4a8d932522629c7325c3"
server_key_iv: "b5e336276e3506268c41a187d7e1157f"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.749017 19245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:16.751842 19259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.753190 19245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:16.753309 19245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "de3c4bd1d90b41449fb456652db19afa"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "0a103ce02e7b4a8d932522629c7325c3"
server_key_iv: "b5e336276e3506268c41a187d7e1157f"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.753419 19245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:16.768994 19245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:16.769717 19245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:16.769897 19245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:16.770463 19245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:16.771490 19245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:16.771539 19245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.771602 19245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:16.771627 19245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.780377 19245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34649
I20260924 07:15:16.780397 19372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34649 every 8 connection(s)
I20260924 07:15:16.781275 19245 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:16.790174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19245
I20260924 07:15:16.790302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:16.790591 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.783006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46313 (local address 127.0.98.62:45635)
0924 07:15:16.783170 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.783175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.784236 (+ 1061us) server_negotiation.cc:407] Connection header received
0924 07:15:16.785621 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.785624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.785675 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.785754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:16.786736 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.787331 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.788539 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.788771 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.788853 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.789941 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:16.789945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:16.789948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:16.789972 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:16.789998 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:16.790009 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:16.790090 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.790358 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.790426 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:15:16.790632 392 external_mini_cluster.cc:1428] Setting key 203a16ca045160a7b90f0848b6590fe9
I20260924 07:15:16.791568 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.783437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45635 (local address 127.0.98.2:46313)
0924 07:15:16.784093 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.784133 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:16.785390 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:16.785961 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:16.785968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:16.786064 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:16.786592 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.786606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.787469 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.787472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.788420 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.788429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.788894 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.788897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.789040 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.789638 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:16.789640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:16.789680 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:16.789748 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:16.790133 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:16.790142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:16.790146 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:16.790265 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 07:15:16.790552 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260924 07:15:16.792850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--builtin_ntp_servers=127.0.98.20:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:16.793041 19373 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45635
I20260924 07:15:16.793284 19373 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:16.793808 19373 heartbeater.cc:511] Master 127.0.98.62:45635 requested a full tablet report, sending...
I20260924 07:15:16.794976 19054 ts_manager.cc:194] Registered new tserver with Master: de3c4bd1d90b41449fb456652db19afa (127.0.98.2:34649)
I20260924 07:15:16.795606 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46313
W20260924 07:15:16.903640 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:16.903919 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:16.903964 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:16.908144 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:16.908205 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:16.908277 19376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:16.913249 19376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:16.914324 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:16.915717 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:16.925372 19376 server_base.cc:1034] running on GCE node
W20260924 07:15:16.925414 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:16.925316 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:16.925321 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:16.925993 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:16.926625 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:16.927817 19376 hybrid_clock.cc:648] HybridClock initialized: now 1790234116927796 us; error 34 us; skew 500 ppm
I20260924 07:15:16.929917 19376 webserver.cc:460] Webserver started at http://127.0.98.3:46037/ using document root <none> and password file <none>
I20260924 07:15:16.930500 19376 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:16.930565 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:16.930748 19376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:16.932886 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "73ac1dc893d34a94bd0b5cdb173279ea"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "4936ec7bf11e89dc4e1b1235d8abb5db"
server_key_iv: "8c99fec39addd31f4d2325bacbc089c0"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.933454 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73ac1dc893d34a94bd0b5cdb173279ea"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "4936ec7bf11e89dc4e1b1235d8abb5db"
server_key_iv: "8c99fec39addd31f4d2325bacbc089c0"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.937278 19376 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:16.939821 19390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.941187 19376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:16.941301 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "73ac1dc893d34a94bd0b5cdb173279ea"
format_stamp: "Formatted at 2026-09-24 07:15:16 on dist-test-slave-cnv3"
server_key: "4936ec7bf11e89dc4e1b1235d8abb5db"
server_key_iv: "8c99fec39addd31f4d2325bacbc089c0"
server_key_version: "encryptionkey@0"
I20260924 07:15:16.941411 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:16.953383 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:16.954101 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:16.954277 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:16.954876 19376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:16.955927 19376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:16.955978 19376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.956032 19376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:16.956058 19376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:16.965204 19376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44125
I20260924 07:15:16.965234 19503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44125 every 8 connection(s)
I20260924 07:15:16.966097 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:16.969403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19376
I20260924 07:15:16.969501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:16.969722 392 external_mini_cluster.cc:1428] Setting key 631cc651db34a3f66431381ff2819ff1
I20260924 07:15:16.976307 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.967925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59843 (local address 127.0.98.62:45635)
0924 07:15:16.968140 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.968145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.969223 (+ 1078us) server_negotiation.cc:407] Connection header received
0924 07:15:16.970835 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.970838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.970893 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.971059 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:16.972282 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.972874 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.974213 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.974445 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.974555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.975775 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:16.975779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:16.975781 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:16.975799 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:16.975819 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:16.975828 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:16.975889 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.976118 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.976182 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:15:16.977203 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.968347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45635 (local address 127.0.98.3:59843)
0924 07:15:16.969067 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:16.969110 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:16.970562 (+ 1452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:16.971280 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:16.971290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:16.971432 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:16.972156 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.972172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.973011 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.973014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.974064 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:16.974076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.974634 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.974637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:16.974792 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.975465 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:16.975467 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:16.975528 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:16.975602 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:16.975932 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:16.975942 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:16.975948 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:15:16.976073 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:15:16.976332 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260924 07:15:16.978487 19504 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45635
I20260924 07:15:16.978740 19504 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:16.979409 19504 heartbeater.cc:511] Master 127.0.98.62:45635 requested a full tablet report, sending...
I20260924 07:15:16.980659 19054 ts_manager.cc:194] Registered new tserver with Master: 73ac1dc893d34a94bd0b5cdb173279ea (127.0.98.3:44125)
I20260924 07:15:16.981256 19054 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59843
I20260924 07:15:16.984098 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:16.989676 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:16.985903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.0.98.62:45635)
0924 07:15:16.986034 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:16.986038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:16.986266 (+ 228us) server_negotiation.cc:407] Connection header received
0924 07:15:16.986330 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:16.986333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:16.986377 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:16.986446 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:16.986667 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.987197 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.988077 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:16.988297 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:16.988382 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:16.989184 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:16.989187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:16.989189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:16.989201 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:16.989221 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:16.989229 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:16.989326 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:16.989461 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 07:15:16.989497 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:16.992825 19054 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:16.995947 19054 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:17.010275 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.004862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44125 (local address 127.0.0.1:55710)
0924 07:15:17.005588 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.005600 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:17.005668 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:17.006256 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:17.006262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:17.006283 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:17.006458 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.006466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.007511 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.007515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.008683 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.008694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.009714 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.009717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.009824 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.009862 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:15:17.009983 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:15:17.010123 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":30,"spinlock_wait_cycles":8576,"thread_start_us":110,"threads_started":1}
I20260924 07:15:17.010279 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.005274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.98.3:44125)
0924 07:15:17.005773 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.005783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.005807 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:15:17.005880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.005885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.006097 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.006252 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:17.006589 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.007414 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.008824 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.009612 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.009714 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.009920 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.010013 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.010111 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":119,"threads_started":1}
I20260924 07:15:17.010607 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.004677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42569 (local address 127.0.0.1:57330)
0924 07:15:17.005242 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.005256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:17.005387 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:17.006299 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:17.006303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:17.006332 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:17.006459 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.006467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.008036 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.008040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.009211 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.009221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.010206 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.010210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.010301 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.010323 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:15:17.010373 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:15:17.010445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":67,"spinlock_wait_cycles":1536,"thread_start_us":105,"threads_started":1}
I20260924 07:15:17.010758 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.004490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34649 (local address 127.0.0.1:44784)
0924 07:15:17.005199 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.005252 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:17.005501 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:17.006533 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:17.006536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:17.006554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:17.006641 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.006647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.008060 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.008062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.009151 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.009162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.010406 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.010409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.010486 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.010506 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:15:17.010575 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:15:17.010648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260924 07:15:17.011137 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.004635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.2:34649)
0924 07:15:17.005913 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.005923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.005942 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:15:17.006153 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.006158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.006388 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.006532 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:17.006732 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.007912 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.009273 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.010069 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.010175 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.010352 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.010627 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.010969 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":133,"threads_started":1}
I20260924 07:15:17.011327 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.005261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.98.1:42569)
0924 07:15:17.005836 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.005849 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.005873 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:15:17.005947 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.005952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.006151 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.006592 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:17.006732 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.007912 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.009306 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.010070 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.010959 (+ 889us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.011094 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.011139 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.011208 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":110,"threads_started":1}
I20260924 07:15:17.013857 19176 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e (DEFAULT_TABLE table=test-table [id=10062da113674e5ca2d0ac379cf4c3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:17.013970 19307 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e (DEFAULT_TABLE table=test-table [id=10062da113674e5ca2d0ac379cf4c3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:17.013970 19438 tablet_service.cc:1467] Processing CreateTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e (DEFAULT_TABLE table=test-table [id=10062da113674e5ca2d0ac379cf4c3e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:17.014966 19176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:17.014973 19307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:17.014973 19438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:17.021392 19521 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Bootstrap starting.
I20260924 07:15:17.023147 19522 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Bootstrap starting.
I20260924 07:15:17.023803 19521 tablet_bootstrap.cc:654] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:17.023867 19523 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Bootstrap starting.
I20260924 07:15:17.024716 19521 log.cc:824] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:17.025445 19522 tablet_bootstrap.cc:654] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:17.025918 19523 tablet_bootstrap.cc:654] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:17.026304 19522 log.cc:824] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:17.026620 19521 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: No bootstrap required, opened a new log
I20260924 07:15:17.026692 19523 log.cc:824] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:17.026820 19521 ts_tablet_manager.cc:1397] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:15:17.028200 19522 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: No bootstrap required, opened a new log
I20260924 07:15:17.028383 19522 ts_tablet_manager.cc:1397] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:15:17.030531 19521 raft_consensus.cc:348] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.030771 19521 raft_consensus.cc:374] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:17.030805 19521 raft_consensus.cc:729] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a85fde65964a38bffd4de8e7beb9e8, State: Initialized, Role: FOLLOWER
I20260924 07:15:17.031260 19521 consensus_queue.cc:260] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.032002 19242 heartbeater.cc:503] Master 127.0.98.62:45635 was elected leader, sending a full tablet report...
I20260924 07:15:17.032076 19522 raft_consensus.cc:348] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.032306 19522 raft_consensus.cc:374] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:17.032333 19521 ts_tablet_manager.cc:1428] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:15:17.032356 19522 raft_consensus.cc:729] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de3c4bd1d90b41449fb456652db19afa, State: Initialized, Role: FOLLOWER
I20260924 07:15:17.032802 19522 consensus_queue.cc:260] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.033579 19373 heartbeater.cc:503] Master 127.0.98.62:45635 was elected leader, sending a full tablet report...
I20260924 07:15:17.033843 19522 ts_tablet_manager.cc:1428] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:15:17.034145 19523 tablet_bootstrap.cc:492] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: No bootstrap required, opened a new log
I20260924 07:15:17.034304 19523 ts_tablet_manager.cc:1397] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 07:15:17.038311 19523 raft_consensus.cc:348] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.038566 19523 raft_consensus.cc:374] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:17.038604 19523 raft_consensus.cc:729] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73ac1dc893d34a94bd0b5cdb173279ea, State: Initialized, Role: FOLLOWER
I20260924 07:15:17.039017 19523 consensus_queue.cc:260] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.039711 19504 heartbeater.cc:503] Master 127.0.98.62:45635 was elected leader, sending a full tablet report...
I20260924 07:15:17.039983 19523 ts_tablet_manager.cc:1428] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 07:15:17.101511 19243 tablet.cc:2369] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:17.169761 19529 raft_consensus.cc:482] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:17.169965 19529 raft_consensus.cc:504] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.171190 19529 leader_election.cc:288] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de3c4bd1d90b41449fb456652db19afa (127.0.98.2:34649), 76a85fde65964a38bffd4de8e7beb9e8 (127.0.98.1:42569)
I20260924 07:15:17.176266 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.171569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53423 (local address 127.0.98.2:34649)
0924 07:15:17.171738 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.171742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.171761 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:15:17.171826 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.171829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.171885 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.171957 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:17.172341 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.173061 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.174885 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.175574 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.175678 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.175741 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.176046 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.176125 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 07:15:17.176266 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.171691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47487 (local address 127.0.98.1:42569)
0924 07:15:17.171815 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.171819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.172237 (+ 418us) server_negotiation.cc:407] Connection header received
0924 07:15:17.172304 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.172328 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.172382 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.172465 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:17.172725 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.173382 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.174941 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.175559 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.175681 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.175741 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.176046 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.176125 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:15:17.176281 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.171448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34649 (local address 127.0.98.3:53423)
0924 07:15:17.171598 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.171616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:17.171736 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:17.172012 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:17.172015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:17.172036 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:17.172214 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.172223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.173246 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.173249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.174715 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.174728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.175697 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.175701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.175870 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.175935 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 07:15:17.176011 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:15:17.176113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":124}
I20260924 07:15:17.176288 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.171499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42569 (local address 127.0.98.3:47487)
0924 07:15:17.172122 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.172137 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:17.172228 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:17.172484 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:17.172486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:17.172502 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:17.172612 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.172625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.173482 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.173486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.174725 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:17.174731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.175699 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.175702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:17.175900 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.175934 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:15:17.176011 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:15:17.176113 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":174,"thread_start_us":130,"threads_started":1}
I20260924 07:15:17.177016 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7e0e0253f4f8881bcb7ca0cfafe2e" candidate_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a85fde65964a38bffd4de8e7beb9e8" is_pre_election: true
I20260924 07:15:17.177021 19327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7e0e0253f4f8881bcb7ca0cfafe2e" candidate_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3c4bd1d90b41449fb456652db19afa" is_pre_election: true
I20260924 07:15:17.177330 19196 raft_consensus.cc:2454] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ac1dc893d34a94bd0b5cdb173279ea in term 0.
I20260924 07:15:17.177372 19327 raft_consensus.cc:2454] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73ac1dc893d34a94bd0b5cdb173279ea in term 0.
I20260924 07:15:17.177824 19392 leader_election.cc:302] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ac1dc893d34a94bd0b5cdb173279ea, 76a85fde65964a38bffd4de8e7beb9e8; no voters:
I20260924 07:15:17.178095 19529 raft_consensus.cc:2781] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:17.178202 19529 raft_consensus.cc:482] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:17.178252 19529 raft_consensus.cc:3037] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:17.179342 19529 raft_consensus.cc:504] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.179749 19529 leader_election.cc:288] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [CANDIDATE]: Term 1 election: Requested vote from peers de3c4bd1d90b41449fb456652db19afa (127.0.98.2:34649), 76a85fde65964a38bffd4de8e7beb9e8 (127.0.98.1:42569)
I20260924 07:15:17.180119 19196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7e0e0253f4f8881bcb7ca0cfafe2e" candidate_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a85fde65964a38bffd4de8e7beb9e8"
I20260924 07:15:17.180119 19327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2af7e0e0253f4f8881bcb7ca0cfafe2e" candidate_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de3c4bd1d90b41449fb456652db19afa"
I20260924 07:15:17.180270 19196 raft_consensus.cc:3037] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:17.180287 19327 raft_consensus.cc:3037] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:17.181524 19327 raft_consensus.cc:2454] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ac1dc893d34a94bd0b5cdb173279ea in term 1.
I20260924 07:15:17.181524 19196 raft_consensus.cc:2454] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73ac1dc893d34a94bd0b5cdb173279ea in term 1.
I20260924 07:15:17.181849 19392 leader_election.cc:302] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73ac1dc893d34a94bd0b5cdb173279ea, 76a85fde65964a38bffd4de8e7beb9e8; no voters:
I20260924 07:15:17.182000 19529 raft_consensus.cc:2781] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:17.182222 19529 raft_consensus.cc:686] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 1 LEADER]: Becoming Leader. State: Replica: 73ac1dc893d34a94bd0b5cdb173279ea, State: Running, Role: LEADER
I20260924 07:15:17.182595 19529 consensus_queue.cc:237] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } }
I20260924 07:15:17.186026 19053 catalog_manager.cc:5495] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 73ac1dc893d34a94bd0b5cdb173279ea (127.0.98.3). New cstate: current_term: 1 leader_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73ac1dc893d34a94bd0b5cdb173279ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44125 } health_report { overall_health: HEALTHY } } }
I20260924 07:15:17.205837 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.202033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.0.98.3:44125)
0924 07:15:17.202162 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:17.202165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:17.202180 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:15:17.202424 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:17.202427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:17.202495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:17.202614 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:17.202862 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.203433 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.204259 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:17.204481 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:17.204559 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:17.204922 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:17.205371 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:17.205511 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:17.205668 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 07:15:17.205702 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:15:17.212126 19327 raft_consensus.cc:1261] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 1 FOLLOWER]: Refusing update from remote peer 73ac1dc893d34a94bd0b5cdb173279ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:17.212126 19196 raft_consensus.cc:1261] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 76a85fde65964a38bffd4de8e7beb9e8 [term 1 FOLLOWER]: Refusing update from remote peer 73ac1dc893d34a94bd0b5cdb173279ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:17.212893 19531 consensus_queue.cc:1035] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a85fde65964a38bffd4de8e7beb9e8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:17.213172 19529 consensus_queue.cc:1035] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "de3c4bd1d90b41449fb456652db19afa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:15:17.216980 19505 tablet.cc:2369] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:17.229806 19054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:17.229977 19054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:17.232391 19054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f92df21d61d48a6b1e4be14e79c7a5f: Sending DeleteTablet for 3 replicas of tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e
I20260924 07:15:17.233197 19307 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:17 UTC) from {username='slave'} at 127.0.0.1:44784
I20260924 07:15:17.233587 19438 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:17 UTC) from {username='slave'} at 127.0.0.1:55710
I20260924 07:15:17.233767 19176 tablet_service.cc:1514] Processing DeleteTablet for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:17 UTC) from {username='slave'} at 127.0.0.1:57330
I20260924 07:15:17.234169 19544 tablet_replica.cc:330] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: stopping tablet replica
I20260924 07:15:17.234249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19114
I20260924 07:15:17.234440 19543 tablet_replica.cc:330] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: stopping tablet replica
I20260924 07:15:17.234517 19544 raft_consensus.cc:2229] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:15:17.234831 19543 raft_consensus.cc:2229] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:17.235008 19544 raft_consensus.cc:2258] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:17.235266 19543 raft_consensus.cc:2258] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:17.236706 19544 ts_tablet_manager.cc:1905] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:17.236737 19543 ts_tablet_manager.cc:1905] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:17.239524 19544 ts_tablet_manager.cc:1918] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:17.239524 19543 ts_tablet_manager.cc:1918] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:17.239625 19544 log.cc:1196] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/2af7e0e0253f4f8881bcb7ca0cfafe2e
I20260924 07:15:17.239774 19543 log.cc:1196] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/2af7e0e0253f4f8881bcb7ca0cfafe2e
I20260924 07:15:17.239912 19544 ts_tablet_manager.cc:1939] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P 73ac1dc893d34a94bd0b5cdb173279ea: Deleting consensus metadata
I20260924 07:15:17.240160 19543 ts_tablet_manager.cc:1939] T 2af7e0e0253f4f8881bcb7ca0cfafe2e P de3c4bd1d90b41449fb456652db19afa: Deleting consensus metadata
I20260924 07:15:17.241348 19041 catalog_manager.cc:4841] TS de3c4bd1d90b41449fb456652db19afa (127.0.98.2:34649): tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e (table test-table [id=10062da113674e5ca2d0ac379cf4c3e9]) successfully deleted
I20260924 07:15:17.241848 19038 catalog_manager.cc:4841] TS 73ac1dc893d34a94bd0b5cdb173279ea (127.0.98.3:44125): tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e (table test-table [id=10062da113674e5ca2d0ac379cf4c3e9]) successfully deleted
W20260924 07:15:17.244104 19039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42569 (error 108)
I20260924 07:15:17.244824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19245
I20260924 07:15:17.245254 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:17.244896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42569 (local address 127.0.0.1:57336)
0924 07:15:17.245088 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:17.245154 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 07:15:17.245766 19039 catalog_manager.cc:4584] TS 76a85fde65964a38bffd4de8e7beb9e8 (127.0.98.1:42569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2af7e0e0253f4f8881bcb7ca0cfafe2e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42569: connect: Connection refused (error 111)
I20260924 07:15:17.250278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19376
I20260924 07:15:17.255573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19023
2026-09-24T07:15:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1104 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:17 dist-test-slave-cnv3 krb5kdc[19549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:17 dist-test-slave-cnv3 krb5kdc[19549](info): set up 2 sockets
Sep 24 07:15:17 dist-test-slave-cnv3 krb5kdc[19549](info): commencing operation
krb5kdc: starting...
W20260924 07:15:18.215293 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:18.343946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:18.456225 19562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:18.456532 19562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:18.456594 19562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:18.460776 19562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:18.460848 19562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:18.460875 19562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:18.460896 19562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:18.460916 19562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:18.466609 19562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:18.467787 19562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:18.468819 19562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:18.477716 19562 server_base.cc:1034] running on GCE node
W20260924 07:15:18.477666 19570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.477723 19567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.477666 19568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:18.478581 19562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:18.479812 19562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:18.480996 19562 hybrid_clock.cc:648] HybridClock initialized: now 1790234118480970 us; error 40 us; skew 500 ppm
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:18.483963 19562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:18.484911 19562 webserver.cc:460] Webserver started at http://127.0.98.62:43571/ using document root <none> and password file <none>
I20260924 07:15:18.485455 19562 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:18.485517 19562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:18.485713 19562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:18.487763 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "6bb9b14d4d5c4b8f882c437134c11dc3"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "0d804f475639d1e5d7913b2a302eabf1"
server_key_iv: "f42c051d46a260edb0bb1e540454e254"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.488291 19562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bb9b14d4d5c4b8f882c437134c11dc3"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "0d804f475639d1e5d7913b2a302eabf1"
server_key_iv: "f42c051d46a260edb0bb1e540454e254"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.492059 19562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:15:18.494684 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:18.496101 19562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:18.496240 19562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "6bb9b14d4d5c4b8f882c437134c11dc3"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "0d804f475639d1e5d7913b2a302eabf1"
server_key_iv: "f42c051d46a260edb0bb1e540454e254"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.496363 19562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:18.512674 19562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:18.515841 19562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:18.516134 19562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:18.523404 19562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34749
I20260924 07:15:18.523442 19629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34749 every 8 connection(s)
I20260924 07:15:18.524515 19562 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:18.527237 19630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:18.530532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19562
I20260924 07:15:18.530642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:18.530900 392 external_mini_cluster.cc:1428] Setting key 27aa656d7c13fbcffdbb11001a0481db
I20260924 07:15:18.532905 19630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: Bootstrap starting.
I20260924 07:15:18.535274 19630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:18.536017 19630 log.cc:824] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:18.537770 19630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: No bootstrap required, opened a new log
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:18.541851 19630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260924 07:15:18.542124 19630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:18.542182 19630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb9b14d4d5c4b8f882c437134c11dc3, State: Initialized, Role: FOLLOWER
I20260924 07:15:18.542649 19630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260924 07:15:18.542824 19630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:18.542892 19630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:18.542989 19630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:18.543910 19630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260924 07:15:18.544260 19630 leader_election.cc:302] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bb9b14d4d5c4b8f882c437134c11dc3; no voters:
I20260924 07:15:18.544543 19630 leader_election.cc:288] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:18.544674 19635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:18.544876 19635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb9b14d4d5c4b8f882c437134c11dc3, State: Running, Role: LEADER
I20260924 07:15:18.545115 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:18.532351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.0.98.62:34749)
0924 07:15:18.532768 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:18.532783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:18.532818 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:18.533465 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:18.533483 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:18.533812 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:18.534080 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:18.534954 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.536035 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.536913 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.537362 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.537500 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:18.540143 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:18.540148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:18.540161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:18.540194 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:18.541916 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:18.542401 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:18.542407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:18.542413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:18.542558 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:18.542894 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:18.542897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:18.542899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:18.543344 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:18.543560 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:18.543859 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:15:18.544167 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260924 07:15:18.545251 19635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } }
I20260924 07:15:18.545725 19630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:18.546427 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bb9b14d4d5c4b8f882c437134c11dc3. Latest consensus state: current_term: 1 leader_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } } }
I20260924 07:15:18.546626 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:18.548167 19644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:18.551039 19644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:18.553115 19636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bb9b14d4d5c4b8f882c437134c11dc3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34749 } } }
I20260924 07:15:18.553270 19636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:18.557021 19644 catalog_manager.cc:1347] Generated new cluster ID: f2a52e78f5384ae3869701babdf23a4e
I20260924 07:15:18.557081 19644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:18.566109 19644 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:18.567425 19644 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:18.584880 19644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: Generated new TSK 0
I20260924 07:15:18.585685 19644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:15:18.615222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:18.728614 19656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:18.728930 19656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:18.728979 19656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:18.733201 19656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:18.733281 19656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:18.733350 19656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:18.738842 19656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:18.740029 19656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:18.741060 19656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:18.751093 19656 server_base.cc:1034] running on GCE node
W20260924 07:15:18.751013 19662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.751008 19664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.751012 19661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:18.751785 19656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:18.752452 19656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:18.753613 19656 hybrid_clock.cc:648] HybridClock initialized: now 1790234118753585 us; error 42 us; skew 500 ppm
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:18.756276 19656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:15:18.757157 19656 webserver.cc:460] Webserver started at http://127.0.98.1:32845/ using document root <none> and password file <none>
I20260924 07:15:18.757694 19656 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:18.757757 19656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:18.757979 19656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:18.760058 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "aacb1b70f21840d18698d85f5b16231f"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "3175d8bd5bb88543ddf995cd2879fec6"
server_key_iv: "89a60a12af095b87cb8acc21a54b1d7a"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.760579 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aacb1b70f21840d18698d85f5b16231f"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "3175d8bd5bb88543ddf995cd2879fec6"
server_key_iv: "89a60a12af095b87cb8acc21a54b1d7a"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.764380 19656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:18.766832 19671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:18.768153 19656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:15:18.768255 19656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "aacb1b70f21840d18698d85f5b16231f"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "3175d8bd5bb88543ddf995cd2879fec6"
server_key_iv: "89a60a12af095b87cb8acc21a54b1d7a"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.768352 19656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:18.781797 19656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:18.784781 19656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:18.784957 19656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:18.785516 19656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:18.786343 19656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:18.786405 19656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:18.786449 19656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:18.786469 19656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:18.795097 19656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39149
I20260924 07:15:18.795176 19784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39149 every 8 connection(s)
I20260924 07:15:18.796068 19656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:18.802241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19656
I20260924 07:15:18.802381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:18.802661 392 external_mini_cluster.cc:1428] Setting key 1b5ff2977192af69f7d3bfe70253d4ec
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:18.809679 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:18.797889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41871 (local address 127.0.98.62:34749)
0924 07:15:18.798041 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:18.798046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:18.798956 (+ 910us) server_negotiation.cc:407] Connection header received
0924 07:15:18.800247 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:18.800250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:18.800299 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:18.800373 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:18.801601 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.802244 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.803628 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.803877 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.803973 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:18.806337 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:18.806340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:18.806342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:18.806367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:18.807693 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:18.808193 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:18.808196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:18.808197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:18.808275 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:18.808601 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:18.808605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:18.808607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:18.808787 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:18.808882 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:18.809382 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 07:15:18.809515 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:15:18.810518 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:18.798316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.1:41871)
0924 07:15:18.798794 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:18.798841 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:18.800060 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:18.800560 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:18.800567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:18.800960 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:18.801464 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:18.801478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.802431 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.802434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:18.803525 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:18.803536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:18.804027 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:18.804032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:18.804212 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:18.804770 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:18.804788 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:18.806163 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:18.807807 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:18.807813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:18.807831 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:18.808095 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:18.808365 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:18.808368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:18.808370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:18.808514 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:18.808884 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:18.808892 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:18.809149 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:15:18.809321 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 07:15:18.809668 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:15:18.811733 19785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260924 07:15:18.811990 19785 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:18.812502 19785 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:15:18.814056 19594 ts_manager.cc:194] Registered new tserver with Master: aacb1b70f21840d18698d85f5b16231f (127.0.98.1:39149)
I20260924 07:15:18.815624 19594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:15:18.831300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:18.945273 19790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:18.945608 19790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:18.945657 19790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:18.950052 19790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:18.950127 19790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:18.950189 19790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:18.955998 19790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:18.957041 19790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:18.958016 19790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:18.967890 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.968031 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:18.967890 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:18.968037 19790 server_base.cc:1034] running on GCE node
I20260924 07:15:18.968534 19790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:18.969190 19790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:18.970359 19790 hybrid_clock.cc:648] HybridClock initialized: now 1790234118970339 us; error 33 us; skew 500 ppm
Sep 24 07:15:18 dist-test-slave-cnv3 krb5kdc[19549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:18.973330 19790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:15:18.974210 19790 webserver.cc:460] Webserver started at http://127.0.98.2:43077/ using document root <none> and password file <none>
I20260924 07:15:18.974727 19790 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:18.974782 19790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:18.974951 19790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:18.977067 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "a331ef3e137d787c1bd977aa187112d5"
server_key_iv: "923fa3a7d2b37ce0a1cb6249c9e85bec"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.977588 19790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "a331ef3e137d787c1bd977aa187112d5"
server_key_iv: "923fa3a7d2b37ce0a1cb6249c9e85bec"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.981367 19790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:18.983929 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:18.985180 19790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:18.985297 19790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187"
format_stamp: "Formatted at 2026-09-24 07:15:18 on dist-test-slave-cnv3"
server_key: "a331ef3e137d787c1bd977aa187112d5"
server_key_iv: "923fa3a7d2b37ce0a1cb6249c9e85bec"
server_key_version: "encryptionkey@0"
I20260924 07:15:18.985401 19790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:19.001734 19790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:19.005106 19790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:19.005265 19790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:19.005807 19790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:19.006611 19790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:19.006659 19790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.006697 19790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:19.006718 19790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.015276 19790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34381
I20260924 07:15:19.015300 19918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34381 every 8 connection(s)
I20260924 07:15:19.016191 19790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:19.017556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19790
I20260924 07:15:19.017669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:19.017962 392 external_mini_cluster.cc:1428] Setting key 891bc5143957525631f35d80325b38ff
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 07:15:19 dist-test-slave-cnv3 krb5kdc[19549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:19.034747 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.018068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60999 (local address 127.0.98.62:34749)
0924 07:15:19.018220 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.018225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.019155 (+ 930us) server_negotiation.cc:407] Connection header received
0924 07:15:19.020549 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.020552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.020606 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.020677 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:19.021913 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.022502 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.024251 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.024481 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.024563 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.030885 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:19.030889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:19.030891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:19.030922 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:19.032426 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.033045 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.033049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.033050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.033141 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.033512 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.033515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.033516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.033741 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:19.033840 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.034420 (+ 580us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.034566 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:15:19.035676 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.018510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.2:60999)
0924 07:15:19.019003 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.019060 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.020332 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.020854 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.020869 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.021219 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:19.021697 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.021710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.022644 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.022648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.024119 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.024132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.024647 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.024652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.024892 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.028459 (+ 3567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:19.028480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:19.030637 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:19.032581 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:19.032589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:19.032611 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:19.032921 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:19.033230 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:19.033240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:19.033242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:19.033406 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:19.033862 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:19.033870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:19.034153 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.034342 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.034723 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":101,"threads_started":1}
I20260924 07:15:19.036931 19919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260924 07:15:19.037179 19919 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:19.037678 19919 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
I20260924 07:15:19.038873 19594 ts_manager.cc:194] Registered new tserver with Master: 9daa9e69ef0a4e1cba3ff4df0ded0187 (127.0.98.2:34381)
I20260924 07:15:19.039574 19594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:15:19.046756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:19.160099 19924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:19.160418 19924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:19.160482 19924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:19.165150 19924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:19.165227 19924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:19.165314 19924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:19.172483 19924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:19.173998 19924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:19.175627 19924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:19.187034 19932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:19.187044 19929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:19.187031 19930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:19.187561 19924 server_base.cc:1034] running on GCE node
I20260924 07:15:19.188056 19924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:19.188768 19924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:19.189956 19924 hybrid_clock.cc:648] HybridClock initialized: now 1790234119189927 us; error 47 us; skew 500 ppm
Sep 24 07:15:19 dist-test-slave-cnv3 krb5kdc[19549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:19.192843 19924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:15:19.193882 19924 webserver.cc:460] Webserver started at http://127.0.98.3:37279/ using document root <none> and password file <none>
I20260924 07:15:19.194504 19924 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:19.194566 19924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:19.194773 19924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:19.196964 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "c398bd243d474af39615a757fb5aa218"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "344cf7261e2403322dc234c49e6270d6"
server_key_iv: "878bf9ad4f4ac7f5d4deb854a06f3ef2"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.197525 19924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c398bd243d474af39615a757fb5aa218"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "344cf7261e2403322dc234c49e6270d6"
server_key_iv: "878bf9ad4f4ac7f5d4deb854a06f3ef2"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.201567 19924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:19.204111 19939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.205338 19924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:19.205442 19924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "c398bd243d474af39615a757fb5aa218"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "344cf7261e2403322dc234c49e6270d6"
server_key_iv: "878bf9ad4f4ac7f5d4deb854a06f3ef2"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.205562 19924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:19.218292 19924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:19.221319 19924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:19.221496 19924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:19.222165 19924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:19.223037 19924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:19.223086 19924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.223146 19924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:19.223173 19924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.232209 19924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33191
I20260924 07:15:19.232247 20052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33191 every 8 connection(s)
I20260924 07:15:19.233100 19924 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:19.233346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 19924
I20260924 07:15:19.233477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:19.233747 392 external_mini_cluster.cc:1428] Setting key 1e66dd0c340e291807e81eeeb4485afc
Sep 24 07:15:19 dist-test-slave-cnv3 krb5kdc[19549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:19.247083 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.235053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41413 (local address 127.0.98.62:34749)
0924 07:15:19.235198 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.235202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.236290 (+ 1088us) server_negotiation.cc:407] Connection header received
0924 07:15:19.237582 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.237586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.237643 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.237712 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:19.238889 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.239484 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.240664 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.240884 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.240980 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.243455 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:19.243460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:19.243462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:19.243491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:19.244860 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.245436 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.245439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.245441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.245518 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.245831 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.245834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.245836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.246021 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:19.246105 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.246711 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.246881 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:15:19.248107 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.235613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34749 (local address 127.0.98.3:41413)
0924 07:15:19.236154 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.236202 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.237370 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.237889 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.237896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.238261 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:19.238748 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.238761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.239605 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.239608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.240536 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.240547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.241034 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.241038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.241205 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.241782 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:19.241799 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:19.243236 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:19.244984 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:19.244996 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:19.245013 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:19.245343 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:19.245599 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:19.245602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:19.245604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:19.245747 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:19.246117 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:19.246124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:19.246451 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.246647 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.247014 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":133,"threads_started":1}
I20260924 07:15:19.249224 20053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34749
I20260924 07:15:19.249461 20053 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:19.249938 20053 heartbeater.cc:511] Master 127.0.98.62:34749 requested a full tablet report, sending...
I20260924 07:15:19.250921 19594 ts_manager.cc:194] Registered new tserver with Master: c398bd243d474af39615a757fb5aa218 (127.0.98.3:33191)
I20260924 07:15:19.251592 19594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41413
I20260924 07:15:19.259881 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:19.269263 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.261811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54726 (local address 127.0.98.62:34749)
0924 07:15:19.261951 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.261954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.262107 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:15:19.262207 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.262209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.262250 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.262315 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:19.262936 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.263589 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.264786 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.265016 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.265105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.266463 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:19.266467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:19.266470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:19.266495 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:19.267704 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.268085 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.268089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.268090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.268165 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:19.268410 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:19.268413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:19.268415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:19.268681 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:19.268814 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.269012 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.269108 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:15:19.272087 19594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:19.275244 19594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:19.289565 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.283406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.0.98.3:33191)
0924 07:15:19.284545 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.284555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.284576 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:15:19.284648 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.284653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.284842 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.284982 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:19.285280 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286508 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.287996 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.288784 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.288891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.289092 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.289285 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.289393 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1039,"thread_start_us":170,"threads_started":1}
I20260924 07:15:19.289709 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.283580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34381 (local address 127.0.0.1:35822)
0924 07:15:19.284252 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.284266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.284361 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.285163 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.285166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.285180 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:19.285253 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.285258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286673 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.286676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.287794 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.287803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.288900 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.288904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.289065 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.289124 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.289241 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.289571 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":189,"thread_start_us":87,"threads_started":1}
I20260924 07:15:19.289902 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.283220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33191 (local address 127.0.0.1:54168)
0924 07:15:19.283620 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.283674 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.283806 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.284988 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.284994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.285014 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:19.285157 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.285163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286636 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.286639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.287860 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.287871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.288917 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.288921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.289062 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.289124 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.289245 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.289492 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":58,"thread_start_us":96,"threads_started":1}
I20260924 07:15:19.290025 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.284048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.0.98.2:34381)
0924 07:15:19.284362 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.284374 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.284445 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:15:19.284526 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.284531 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.284937 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.285074 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:19.285364 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286508 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.287907 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.288784 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.288891 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.289043 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.289769 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.289869 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260924 07:15:19.290828 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.283982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39149 (local address 127.0.0.1:47980)
0924 07:15:19.284727 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.284739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.284824 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.285285 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.285288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.285306 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:19.285403 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.285410 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286364 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.286367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.287860 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.287871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.290461 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.290465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.290551 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.290575 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.290631 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.290702 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":64,"thread_start_us":79,"threads_started":1}
I20260924 07:15:19.291280 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.284179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.0.98.1:39149)
0924 07:15:19.284597 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.284609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.284926 (+ 317us) server_negotiation.cc:407] Connection header received
0924 07:15:19.285003 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.285008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.285173 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.285301 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:19.285489 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.286250 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.289126 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.289929 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.290033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.291014 (+ 981us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.291064 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.291144 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":100,"threads_started":1}
I20260924 07:15:19.293195 19987 tablet_service.cc:1467] Processing CreateTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 (DEFAULT_TABLE table=test-table [id=e478d49d053e4f46bb969e8799db3104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:19.293195 19853 tablet_service.cc:1467] Processing CreateTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 (DEFAULT_TABLE table=test-table [id=e478d49d053e4f46bb969e8799db3104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:19.293589 19719 tablet_service.cc:1467] Processing CreateTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 (DEFAULT_TABLE table=test-table [id=e478d49d053e4f46bb969e8799db3104]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:19.294351 19853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dd1f897e8470cb0d64aecfc88ae05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:19.294351 19987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dd1f897e8470cb0d64aecfc88ae05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:19.294472 19719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5dd1f897e8470cb0d64aecfc88ae05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:19.301133 20070 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: Bootstrap starting.
I20260924 07:15:19.302279 20072 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Bootstrap starting.
I20260924 07:15:19.302481 20071 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Bootstrap starting.
I20260924 07:15:19.303512 20070 tablet_bootstrap.cc:654] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:19.304534 20070 log.cc:824] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:19.304566 20071 tablet_bootstrap.cc:654] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:19.304566 20072 tablet_bootstrap.cc:654] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:19.305785 20072 log.cc:824] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:19.305785 20071 log.cc:824] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:19.306826 20070 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: No bootstrap required, opened a new log
I20260924 07:15:19.307091 20070 ts_tablet_manager.cc:1397] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:15:19.307874 20071 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: No bootstrap required, opened a new log
I20260924 07:15:19.308080 20071 ts_tablet_manager.cc:1397] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:15:19.308176 20072 tablet_bootstrap.cc:492] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: No bootstrap required, opened a new log
I20260924 07:15:19.308349 20072 ts_tablet_manager.cc:1397] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:15:19.310379 20070 raft_consensus.cc:348] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.310621 20070 raft_consensus.cc:374] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:19.310691 20070 raft_consensus.cc:729] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacb1b70f21840d18698d85f5b16231f, State: Initialized, Role: FOLLOWER
I20260924 07:15:19.311131 20070 consensus_queue.cc:260] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.312081 20070 ts_tablet_manager.cc:1428] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260924 07:15:19.312165 20072 raft_consensus.cc:348] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.312171 20071 raft_consensus.cc:348] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.312443 20071 raft_consensus.cc:374] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:19.312479 19785 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260924 07:15:19.312523 20071 raft_consensus.cc:729] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9daa9e69ef0a4e1cba3ff4df0ded0187, State: Initialized, Role: FOLLOWER
I20260924 07:15:19.312608 20072 raft_consensus.cc:374] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:19.312654 20072 raft_consensus.cc:729] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c398bd243d474af39615a757fb5aa218, State: Initialized, Role: FOLLOWER
I20260924 07:15:19.312997 20071 consensus_queue.cc:260] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.313017 20072 consensus_queue.cc:260] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.313666 19919 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260924 07:15:19.313895 20053 heartbeater.cc:503] Master 127.0.98.62:34749 was elected leader, sending a full tablet report...
I20260924 07:15:19.313998 20072 ts_tablet_manager.cc:1428] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:15:19.314870 20071 ts_tablet_manager.cc:1428] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:15:19.346282 20077 raft_consensus.cc:482] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:19.346544 20077 raft_consensus.cc:504] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.347793 20077 leader_election.cc:288] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c398bd243d474af39615a757fb5aa218 (127.0.98.3:33191), aacb1b70f21840d18698d85f5b16231f (127.0.98.1:39149)
I20260924 07:15:19.352388 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.348126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55449 (local address 127.0.98.3:33191)
0924 07:15:19.348255 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.348260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.348278 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:15:19.348337 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.348341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.348393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.348479 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:19.348826 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.349585 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351108 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.351789 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351912 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.351980 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.352185 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.352257 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:15:19.352412 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.348031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33191 (local address 127.0.98.2:55449)
0924 07:15:19.348157 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.348171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.348257 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.348575 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.348578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.348598 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:19.348718 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.348724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.349698 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.349700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.350966 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.350976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.351911 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.352021 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.352083 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.352159 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.352239 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":31}
I20260924 07:15:19.352388 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.348433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40543 (local address 127.0.98.1:39149)
0924 07:15:19.348561 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.348565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.348636 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:15:19.348730 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.348732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.348771 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.348851 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:19.349079 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.349825 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351171 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.351851 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351940 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.351991 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.352185 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.352257 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:15:19.352420 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.348295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39149 (local address 127.0.98.2:40543)
0924 07:15:19.348563 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.348574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.348653 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.348870 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.348872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.348888 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:19.348988 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.348992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.349922 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.349925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.351055 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.351062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.351945 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.351948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.352067 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.352086 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.352154 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.352239 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":7,"thread_start_us":123,"threads_started":1}
I20260924 07:15:19.353108 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dd1f897e8470cb0d64aecfc88ae05" candidate_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c398bd243d474af39615a757fb5aa218" is_pre_election: true
I20260924 07:15:19.353410 20007 raft_consensus.cc:2454] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9daa9e69ef0a4e1cba3ff4df0ded0187 in term 0.
I20260924 07:15:19.353381 19739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dd1f897e8470cb0d64aecfc88ae05" candidate_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacb1b70f21840d18698d85f5b16231f" is_pre_election: true
I20260924 07:15:19.353621 19739 raft_consensus.cc:2454] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9daa9e69ef0a4e1cba3ff4df0ded0187 in term 0.
I20260924 07:15:19.353860 19807 leader_election.cc:302] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9daa9e69ef0a4e1cba3ff4df0ded0187, c398bd243d474af39615a757fb5aa218; no voters:
I20260924 07:15:19.354117 20077 raft_consensus.cc:2781] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:19.354195 20077 raft_consensus.cc:482] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:19.354238 20077 raft_consensus.cc:3037] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:19.355307 20077 raft_consensus.cc:504] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.355666 20077 leader_election.cc:288] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [CANDIDATE]: Term 1 election: Requested vote from peers c398bd243d474af39615a757fb5aa218 (127.0.98.3:33191), aacb1b70f21840d18698d85f5b16231f (127.0.98.1:39149)
I20260924 07:15:19.356103 20007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dd1f897e8470cb0d64aecfc88ae05" candidate_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c398bd243d474af39615a757fb5aa218"
I20260924 07:15:19.356115 19739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5dd1f897e8470cb0d64aecfc88ae05" candidate_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacb1b70f21840d18698d85f5b16231f"
I20260924 07:15:19.356223 19739 raft_consensus.cc:3037] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:19.356246 20007 raft_consensus.cc:3037] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:19.357241 19739 raft_consensus.cc:2454] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9daa9e69ef0a4e1cba3ff4df0ded0187 in term 1.
I20260924 07:15:19.357338 20007 raft_consensus.cc:2454] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9daa9e69ef0a4e1cba3ff4df0ded0187 in term 1.
I20260924 07:15:19.357558 19807 leader_election.cc:302] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9daa9e69ef0a4e1cba3ff4df0ded0187, aacb1b70f21840d18698d85f5b16231f; no voters:
I20260924 07:15:19.357725 20077 raft_consensus.cc:2781] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:19.357959 20077 raft_consensus.cc:686] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 1 LEADER]: Becoming Leader. State: Replica: 9daa9e69ef0a4e1cba3ff4df0ded0187, State: Running, Role: LEADER
I20260924 07:15:19.358312 20077 consensus_queue.cc:237] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } }
I20260924 07:15:19.361631 19593 catalog_manager.cc:5495] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9daa9e69ef0a4e1cba3ff4df0ded0187 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9daa9e69ef0a4e1cba3ff4df0ded0187" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34381 } health_report { overall_health: HEALTHY } } }
I20260924 07:15:19.389209 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.385512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.0.98.2:34381)
0924 07:15:19.385647 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.385651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.385665 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:15:19.385785 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.385787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.385840 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.385925 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:19.386174 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.386696 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.387575 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.387892 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.387992 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.388371 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:19.388799 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:19.388890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.388961 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.389018 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:15:19.395607 19739 raft_consensus.cc:1261] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 1 FOLLOWER]: Refusing update from remote peer 9daa9e69ef0a4e1cba3ff4df0ded0187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:19.395608 20007 raft_consensus.cc:1261] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 1 FOLLOWER]: Refusing update from remote peer 9daa9e69ef0a4e1cba3ff4df0ded0187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:19.396409 20080 consensus_queue.cc:1035] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aacb1b70f21840d18698d85f5b16231f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:19.396639 20077 consensus_queue.cc:1035] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c398bd243d474af39615a757fb5aa218" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:19.408069 20083 mvcc.cc:205] Tried to move back new op lower bound from 7332798953036120064 to 7332798952894328832. Current Snapshot: MvccSnapshot[applied={T|T < 7332798953036120064 or (T in {7332798953036120064})}]
I20260924 07:15:19.409883 20085 mvcc.cc:205] Tried to move back new op lower bound from 7332798953036120064 to 7332798952894328832. Current Snapshot: MvccSnapshot[applied={T|T < 7332798953036120064}]
I20260924 07:15:19.410107 20084 mvcc.cc:205] Tried to move back new op lower bound from 7332798953036120064 to 7332798952894328832. Current Snapshot: MvccSnapshot[applied={T|T < 7332798953036120064 or (T in {7332798953036120064})}]
I20260924 07:15:19.415585 19593 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:19.415746 19593 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:19.418026 19593 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bb9b14d4d5c4b8f882c437134c11dc3: Sending DeleteTablet for 3 replicas of tablet de5dd1f897e8470cb0d64aecfc88ae05
I20260924 07:15:19.418889 19719 tablet_service.cc:1514] Processing DeleteTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47980
I20260924 07:15:19.418893 19853 tablet_service.cc:1514] Processing DeleteTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35822
I20260924 07:15:19.419713 20094 tablet_replica.cc:330] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f: stopping tablet replica
I20260924 07:15:19.419711 19987 tablet_service.cc:1514] Processing DeleteTablet for tablet de5dd1f897e8470cb0d64aecfc88ae05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54168
I20260924 07:15:19.419799 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19656
I20260924 07:15:19.420050 20094 raft_consensus.cc:2229] T de5dd1f897e8470cb0d64aecfc88ae05 P aacb1b70f21840d18698d85f5b16231f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:19.420442 20093 tablet_replica.cc:330] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: stopping tablet replica
I20260924 07:15:19.420719 20093 raft_consensus.cc:2229] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:15:19.421072 20093 raft_consensus.cc:2258] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:19.421118 20095 tablet_replica.cc:330] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: stopping tablet replica
I20260924 07:15:19.421489 20095 raft_consensus.cc:2229] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:19.421831 20095 raft_consensus.cc:2258] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:19.423502 20095 ts_tablet_manager.cc:1905] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:19.423607 20093 ts_tablet_manager.cc:1905] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:19.426349 20095 ts_tablet_manager.cc:1918] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:19.426349 20093 ts_tablet_manager.cc:1918] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:19.426458 20093 log.cc:1196] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/de5dd1f897e8470cb0d64aecfc88ae05
I20260924 07:15:19.426597 20095 log.cc:1196] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/de5dd1f897e8470cb0d64aecfc88ae05
I20260924 07:15:19.426815 20093 ts_tablet_manager.cc:1939] T de5dd1f897e8470cb0d64aecfc88ae05 P 9daa9e69ef0a4e1cba3ff4df0ded0187: Deleting consensus metadata
I20260924 07:15:19.426961 20095 ts_tablet_manager.cc:1939] T de5dd1f897e8470cb0d64aecfc88ae05 P c398bd243d474af39615a757fb5aa218: Deleting consensus metadata
I20260924 07:15:19.428189 19579 catalog_manager.cc:4841] TS c398bd243d474af39615a757fb5aa218 (127.0.98.3:33191): tablet de5dd1f897e8470cb0d64aecfc88ae05 (table test-table [id=e478d49d053e4f46bb969e8799db3104]) successfully deleted
I20260924 07:15:19.428189 19578 catalog_manager.cc:4841] TS 9daa9e69ef0a4e1cba3ff4df0ded0187 (127.0.98.2:34381): tablet de5dd1f897e8470cb0d64aecfc88ae05 (table test-table [id=e478d49d053e4f46bb969e8799db3104]) successfully deleted
I20260924 07:15:19.428506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19790
W20260924 07:15:19.428520 19579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39149 (error 108)
I20260924 07:15:19.430328 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.430005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39149 (local address 127.0.0.1:47990)
0924 07:15:19.430151 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.430228 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 07:15:19.430521 19579 catalog_manager.cc:4584] TS aacb1b70f21840d18698d85f5b16231f (127.0.98.1:39149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de5dd1f897e8470cb0d64aecfc88ae05: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39149: connect: Connection refused (error 111)
I20260924 07:15:19.433950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19924
I20260924 07:15:19.438997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 19562
2026-09-24T07:15:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T07:15:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:19Z Disabled control of system clock
I20260924 07:15:19.467630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37971
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:19.578868 20100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:19.579169 20100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:19.579205 20100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:19.583443 20100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:19.583513 20100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:19.583526 20100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:19.583542 20100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:19.583556 20100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:19.588586 20100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:19.589704 20100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:19.591058 20100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:19.599815 20105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:19.599759 20108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:19.599798 20106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:19.600018 20100 server_base.cc:1034] running on GCE node
I20260924 07:15:19.600800 20100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:19.602057 20100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:19.603246 20100 hybrid_clock.cc:648] HybridClock initialized: now 1790234119603204 us; error 58 us; skew 500 ppm
I20260924 07:15:19.605311 20100 webserver.cc:460] Webserver started at http://127.0.98.62:42821/ using document root <none> and password file <none>
I20260924 07:15:19.605933 20100 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:19.606005 20100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:19.606200 20100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:19.608316 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "4081bdbedc674b308a3a5fee9e0abf63"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "274db830ff9db35ae4e57dc24e8fccd4"
server_key_iv: "35bc81625b0a216376349a735db29124"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.608847 20100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "4081bdbedc674b308a3a5fee9e0abf63"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "274db830ff9db35ae4e57dc24e8fccd4"
server_key_iv: "35bc81625b0a216376349a735db29124"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.612540 20100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:15:19.615232 20114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.616585 20100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:19.616698 20100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "4081bdbedc674b308a3a5fee9e0abf63"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "274db830ff9db35ae4e57dc24e8fccd4"
server_key_iv: "35bc81625b0a216376349a735db29124"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.616847 20100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:19.649681 20100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:19.650444 20100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:19.650630 20100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:19.657688 20100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37971
I20260924 07:15:19.657704 20166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37971 every 8 connection(s)
I20260924 07:15:19.658650 20100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:19.661885 20167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:19.664108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20100
I20260924 07:15:19.664212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:19.664535 392 external_mini_cluster.cc:1428] Setting key 0d67921ad5b79970cecf57e864a5e6fe
I20260924 07:15:19.667955 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: Bootstrap starting.
I20260924 07:15:19.670233 20167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:19.670972 20167 log.cc:824] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:19.672895 20167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: No bootstrap required, opened a new log
I20260924 07:15:19.673816 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.666039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.98.62:37971)
0924 07:15:19.666675 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.666691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.666727 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:19.667421 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.667443 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.667850 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.668158 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:19.668579 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.669533 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.670400 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.670881 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.671022 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.672016 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:19.672022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:19.672036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:19.672431 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:19.672498 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:19.672519 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:19.672608 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.672748 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.673025 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260924 07:15:19.676661 20167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } }
I20260924 07:15:19.676934 20167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:19.676973 20167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4081bdbedc674b308a3a5fee9e0abf63, State: Initialized, Role: FOLLOWER
I20260924 07:15:19.677429 20167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } }
I20260924 07:15:19.677560 20167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:19.677632 20167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:19.677740 20167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:19.678720 20167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } }
I20260924 07:15:19.679044 20167 leader_election.cc:302] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4081bdbedc674b308a3a5fee9e0abf63; no voters:
I20260924 07:15:19.679353 20167 leader_election.cc:288] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:19.679533 20172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:19.679736 20172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [term 1 LEADER]: Becoming Leader. State: Replica: 4081bdbedc674b308a3a5fee9e0abf63, State: Running, Role: LEADER
I20260924 07:15:19.680076 20172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } }
I20260924 07:15:19.680652 20167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:19.681715 20173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4081bdbedc674b308a3a5fee9e0abf63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } } }
I20260924 07:15:19.681842 20173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:19.682168 20174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4081bdbedc674b308a3a5fee9e0abf63. Latest consensus state: current_term: 1 leader_uuid: "4081bdbedc674b308a3a5fee9e0abf63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4081bdbedc674b308a3a5fee9e0abf63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37971 } } }
I20260924 07:15:19.682298 20174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:19.682214 20181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:19.685608 20181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:19.691462 20181 catalog_manager.cc:1347] Generated new cluster ID: 083fc2ac92b24a28bac2ca8a622631c2
I20260924 07:15:19.691644 20181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:19.709761 20181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:19.711110 20181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:19.722913 20181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: Generated new TSK 0
I20260924 07:15:19.723897 20181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:19.734442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:15:19.846447 20191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:19.846742 20191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:19.846782 20191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:19.850983 20191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:19.851055 20191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:19.851114 20191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:15:19.856122 20191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:19.857254 20191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:19.858685 20191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:19.868322 20197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:19.868315 20196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:19.868367 20191 server_base.cc:1034] running on GCE node
W20260924 07:15:19.868314 20199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:19.869005 20191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:19.869652 20191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:19.870838 20191 hybrid_clock.cc:648] HybridClock initialized: now 1790234119870821 us; error 33 us; skew 500 ppm
I20260924 07:15:19.873052 20191 webserver.cc:460] Webserver started at http://127.0.98.1:35099/ using document root <none> and password file <none>
I20260924 07:15:19.873637 20191 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:19.873700 20191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:19.873908 20191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:19.876032 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data/instance:
uuid: "283a3810524845fba67f609e69f88e92"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "aafe38d91ea7ebb34c81fedf1584b590"
server_key_iv: "9be0bc20efb60d6a014b2b4f59b4e9a8"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.876581 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal/instance:
uuid: "283a3810524845fba67f609e69f88e92"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "aafe38d91ea7ebb34c81fedf1584b590"
server_key_iv: "9be0bc20efb60d6a014b2b4f59b4e9a8"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.880291 20191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:15:19.882836 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.884088 20191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:19.884199 20191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal
uuid: "283a3810524845fba67f609e69f88e92"
format_stamp: "Formatted at 2026-09-24 07:15:19 on dist-test-slave-cnv3"
server_key: "aafe38d91ea7ebb34c81fedf1584b590"
server_key_iv: "9be0bc20efb60d6a014b2b4f59b4e9a8"
server_key_version: "encryptionkey@0"
I20260924 07:15:19.884308 20191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:19.905516 20191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:19.906252 20191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:19.906435 20191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:19.907016 20191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:19.907986 20191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:19.908033 20191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.908077 20191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:19.908107 20191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:19.916536 20191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35399
I20260924 07:15:19.916554 20318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35399 every 8 connection(s)
I20260924 07:15:19.917439 20191 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:15:19.921298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20191
I20260924 07:15:19.921435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:15:19.921743 392 external_mini_cluster.cc:1428] Setting key 80d412f3348dc19966abd4f53fae9fba
I20260924 07:15:19.923758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:19.927857 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.919248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45671 (local address 127.0.98.62:37971)
0924 07:15:19.919395 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:19.919399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:19.920501 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 07:15:19.922367 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:19.922370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:19.922419 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:19.922496 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:19.923593 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.924278 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.925654 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.925998 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.926090 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.927291 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:19.927293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:19.927296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:19.927309 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:19.927329 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:19.927337 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:19.927430 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:19.927647 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 07:15:19.927710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:15:19.928751 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:19.919731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37971 (local address 127.0.98.1:45671)
0924 07:15:19.920370 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:19.920409 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:19.921926 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:19.922673 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:19.922680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:19.922763 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:19.923435 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.923454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.924427 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.924431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.925531 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:19.925541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:19.926154 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:19.926158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:19.926343 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:19.926980 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:19.926982 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:19.927037 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:19.927122 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:19.927454 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:19.927464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:19.927468 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:19.927631 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 07:15:19.927930 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260924 07:15:19.929943 20319 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37971
I20260924 07:15:19.930168 20319 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:19.930671 20319 heartbeater.cc:511] Master 127.0.98.62:37971 requested a full tablet report, sending...
I20260924 07:15:19.932327 20131 ts_manager.cc:194] Registered new tserver with Master: 283a3810524845fba67f609e69f88e92 (127.0.98.1:35399)
I20260924 07:15:19.933939 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45671
W20260924 07:15:20.035000 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:20.035286 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:20.035323 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:20.039469 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:20.039534 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:20.039594 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:15:20.044581 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:20.045687 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:20.047098 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:20.057791 20328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:20.057796 20330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:20.057796 20327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:20.057859 20322 server_base.cc:1034] running on GCE node
I20260924 07:15:20.058409 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:20.059108 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:20.060294 20322 hybrid_clock.cc:648] HybridClock initialized: now 1790234120060225 us; error 83 us; skew 500 ppm
I20260924 07:15:20.062386 20322 webserver.cc:460] Webserver started at http://127.0.98.2:39763/ using document root <none> and password file <none>
I20260924 07:15:20.062965 20322 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:20.063043 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:20.063248 20322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:20.065428 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data/instance:
uuid: "e38b086fec7c4571b0f0c55de5df1b56"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "33857a891f94d7a8a10584ff543e3bae"
server_key_iv: "4b83348199f23cceac26fef31ccdc8ab"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.065961 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e38b086fec7c4571b0f0c55de5df1b56"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "33857a891f94d7a8a10584ff543e3bae"
server_key_iv: "4b83348199f23cceac26fef31ccdc8ab"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.069562 20322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:15:20.072213 20336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.073597 20322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:20.073724 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal
uuid: "e38b086fec7c4571b0f0c55de5df1b56"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "33857a891f94d7a8a10584ff543e3bae"
server_key_iv: "4b83348199f23cceac26fef31ccdc8ab"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.073843 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:20.118260 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:20.119023 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:20.119218 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:20.119843 20322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:20.120913 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:20.120963 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.121018 20322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:20.121043 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.129868 20322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41821
I20260924 07:15:20.129957 20449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41821 every 8 connection(s)
I20260924 07:15:20.130834 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:15:20.140364 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.132895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56731 (local address 127.0.98.62:37971)
0924 07:15:20.133072 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.133076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.134264 (+ 1188us) server_negotiation.cc:407] Connection header received
0924 07:15:20.135541 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.135544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.135597 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.135672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:20.136585 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.137159 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.138326 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.138558 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.138647 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.139928 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:20.139932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:20.139934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:20.139947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:20.139968 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:20.139976 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:20.140036 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.140179 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.140240 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 07:15:20.141100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20322
I20260924 07:15:20.141224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:15:20.141422 20452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.133296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37971 (local address 127.0.98.2:56731)
0924 07:15:20.133978 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.134021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.135275 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.135830 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.135837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.135922 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:20.136428 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.136440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.137283 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.137286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.138209 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.138218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.138722 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.138725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.138896 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.139651 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:20.139652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:20.139696 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:20.139766 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:20.140056 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:20.140063 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:20.140067 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.140172 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.140416 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260924 07:15:20.141624 392 external_mini_cluster.cc:1428] Setting key 19af50a335befd828b2faed57e141184
I20260924 07:15:20.142683 20450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37971
I20260924 07:15:20.142935 20450 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:20.143572 20450 heartbeater.cc:511] Master 127.0.98.62:37971 requested a full tablet report, sending...
I20260924 07:15:20.143708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--builtin_ntp_servers=127.0.98.20:41073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:15:20.144693 20131 ts_manager.cc:194] Registered new tserver with Master: e38b086fec7c4571b0f0c55de5df1b56 (127.0.98.2:41821)
I20260924 07:15:20.145263 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56731
W20260924 07:15:20.259402 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:20.259727 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:20.259774 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:20.263881 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:20.263949 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:20.264011 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:15:20.268963 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:20.270092 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:20.271531 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:20.281633 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:20.281651 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:20.281651 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:20.281885 20453 server_base.cc:1034] running on GCE node
I20260924 07:15:20.282368 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:20.283082 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:20.284255 20453 hybrid_clock.cc:648] HybridClock initialized: now 1790234120284220 us; error 44 us; skew 500 ppm
I20260924 07:15:20.286211 20453 webserver.cc:460] Webserver started at http://127.0.98.3:43811/ using document root <none> and password file <none>
I20260924 07:15:20.286803 20453 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:20.286877 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:20.287084 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:20.289287 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data/instance:
uuid: "16d0b0a0cbf047c18117987fa6a90f44"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "7e406007e3a422c0723db91d13934269"
server_key_iv: "ac081b84f52c5284f1d73257fdd1e65d"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.289855 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal/instance:
uuid: "16d0b0a0cbf047c18117987fa6a90f44"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "7e406007e3a422c0723db91d13934269"
server_key_iv: "ac081b84f52c5284f1d73257fdd1e65d"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.293674 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:20.296341 20467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.297814 20453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:20.297945 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal
uuid: "16d0b0a0cbf047c18117987fa6a90f44"
format_stamp: "Formatted at 2026-09-24 07:15:20 on dist-test-slave-cnv3"
server_key: "7e406007e3a422c0723db91d13934269"
server_key_iv: "ac081b84f52c5284f1d73257fdd1e65d"
server_key_version: "encryptionkey@0"
I20260924 07:15:20.298090 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:20.312513 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:20.313244 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:20.313427 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:20.314009 20453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:15:20.315033 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:15:20.315090 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.315146 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:15:20.315176 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:20.323930 20453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36987
I20260924 07:15:20.323972 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36987 every 8 connection(s)
I20260924 07:15:20.324883 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:15:20.330835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20453
I20260924 07:15:20.330952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:15:20.331192 392 external_mini_cluster.cc:1428] Setting key 546a4a2dc98e08ea5817933739b96843
I20260924 07:15:20.334735 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.326754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54021 (local address 127.0.98.62:37971)
0924 07:15:20.326932 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.326936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.328046 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 07:15:20.329482 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.329485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.329540 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.329631 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:20.330541 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.331148 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.332604 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.332901 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.332999 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.334181 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:20.334184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:20.334186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:20.334199 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:20.334220 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:20.334228 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:20.334289 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.334560 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.334619 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:15:20.335521 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.327167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37971 (local address 127.0.98.3:54021)
0924 07:15:20.327885 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.327928 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.329183 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.329800 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.329807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.329894 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:20.330390 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.330403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.331288 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.331292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.332497 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.332509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.333094 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.333099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.333302 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.333886 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:15:20.333887 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:20.333930 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:15:20.334002 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:20.334297 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:20.334302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:20.334306 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.334398 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.334653 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":142,"threads_started":1}
I20260924 07:15:20.336702 20581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37971
I20260924 07:15:20.336928 20581 heartbeater.cc:464] Registering TS with master...
I20260924 07:15:20.337455 20581 heartbeater.cc:511] Master 127.0.98.62:37971 requested a full tablet report, sending...
I20260924 07:15:20.338551 20131 ts_manager.cc:194] Registered new tserver with Master: 16d0b0a0cbf047c18117987fa6a90f44 (127.0.98.3:36987)
I20260924 07:15:20.339157 20131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54021
I20260924 07:15:20.345384 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:15:20.351163 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.347540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.0.98.62:37971)
0924 07:15:20.347687 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.347692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.347747 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:15:20.347879 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.347881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.347924 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.347993 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:20.348232 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.348739 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.349684 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.349897 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.349974 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.350706 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:20.350709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:20.350711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:20.350724 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:20.350745 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:20.350753 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:20.350810 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.350935 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.350991 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:15:20.354054 20131 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:15:20.357187 20131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:15:20.371605 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.365798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36987 (local address 127.0.0.1:58354)
0924 07:15:20.367032 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.367045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.367131 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.367710 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.367715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.367737 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:20.367872 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.367879 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.368930 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.368934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.370157 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.370168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.370990 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.370994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.371107 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.371147 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.371252 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.371431 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1175,"mutex_wait_us":519,"thread_start_us":77,"threads_started":1}
I20260924 07:15:20.371767 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.365798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41821 (local address 127.0.0.1:54626)
0924 07:15:20.366858 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.366943 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.367093 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.367859 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.367862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.367875 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:20.367967 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.367973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.369417 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.369421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.370515 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.370523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.371279 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.371283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.371444 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.371470 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.371557 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.371627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"mutex_wait_us":2,"spinlock_wait_cycles":1792,"thread_start_us":97,"threads_started":1}
I20260924 07:15:20.372112 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.365956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.0.98.3:36987)
0924 07:15:20.366699 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.366711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.367232 (+ 521us) server_negotiation.cc:407] Connection header received
0924 07:15:20.367318 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.367323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.367542 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.367692 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:20.367994 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.368809 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.370305 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.370866 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.370964 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.371182 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.371598 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.371708 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":157,"threads_started":1}
I20260924 07:15:20.372186 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.366039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.98.2:41821)
0924 07:15:20.366846 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.366857 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.367232 (+ 375us) server_negotiation.cc:407] Connection header received
0924 07:15:20.367313 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.367318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.367568 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.367713 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:20.368085 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.369288 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.370625 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.371147 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.371256 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.371502 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.371937 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.372042 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":119,"threads_started":1}
I20260924 07:15:20.373090 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.366104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:37930)
0924 07:15:20.366996 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.367008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.367092 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.367883 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.367886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.367927 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:20.368029 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.368036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.369436 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.369439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.371613 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.371623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.372726 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.372730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.372812 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.372833 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.372889 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.372956 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":1045,"spinlock_wait_cycles":896,"thread_start_us":88,"threads_started":1}
I20260924 07:15:20.373196 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.366490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37930 (local address 127.0.98.1:35399)
0924 07:15:20.366901 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.366911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.367232 (+ 321us) server_negotiation.cc:407] Connection header received
0924 07:15:20.367313 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.367318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.367670 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.367787 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:20.368156 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.369288 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.371963 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.372612 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.372714 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.372896 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.372944 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.373032 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":123,"threads_started":1}
I20260924 07:15:20.376020 20384 tablet_service.cc:1467] Processing CreateTablet for tablet 346e945f65054ff69a107ff4c429d8c8 (DEFAULT_TABLE table=test-table [id=f11b3bbef26d4658b84b2e168559d229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:20.376107 20515 tablet_service.cc:1467] Processing CreateTablet for tablet 346e945f65054ff69a107ff4c429d8c8 (DEFAULT_TABLE table=test-table [id=f11b3bbef26d4658b84b2e168559d229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:20.376403 20253 tablet_service.cc:1467] Processing CreateTablet for tablet 346e945f65054ff69a107ff4c429d8c8 (DEFAULT_TABLE table=test-table [id=f11b3bbef26d4658b84b2e168559d229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:15:20.377122 20384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346e945f65054ff69a107ff4c429d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:20.377208 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346e945f65054ff69a107ff4c429d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:20.377264 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 346e945f65054ff69a107ff4c429d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:20.383724 20598 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Bootstrap starting.
I20260924 07:15:20.385001 20600 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Bootstrap starting.
I20260924 07:15:20.386219 20598 tablet_bootstrap.cc:654] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:20.386827 20600 tablet_bootstrap.cc:654] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:20.386966 20598 log.cc:824] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:20.387599 20600 log.cc:824] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:20.388142 20599 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Bootstrap starting.
I20260924 07:15:20.389320 20598 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: No bootstrap required, opened a new log
I20260924 07:15:20.389513 20598 ts_tablet_manager.cc:1397] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:15:20.390326 20599 tablet_bootstrap.cc:654] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:20.390991 20599 log.cc:824] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:20.392652 20600 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: No bootstrap required, opened a new log
I20260924 07:15:20.392846 20600 ts_tablet_manager.cc:1397] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260924 07:15:20.393021 20599 tablet_bootstrap.cc:492] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: No bootstrap required, opened a new log
I20260924 07:15:20.393214 20599 ts_tablet_manager.cc:1397] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:15:20.393493 20598 raft_consensus.cc:348] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.393803 20598 raft_consensus.cc:374] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:20.393889 20598 raft_consensus.cc:729] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38b086fec7c4571b0f0c55de5df1b56, State: Initialized, Role: FOLLOWER
I20260924 07:15:20.394380 20598 consensus_queue.cc:260] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.395138 20450 heartbeater.cc:503] Master 127.0.98.62:37971 was elected leader, sending a full tablet report...
I20260924 07:15:20.395516 20598 ts_tablet_manager.cc:1428] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:15:20.396730 20600 raft_consensus.cc:348] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.397002 20600 raft_consensus.cc:374] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:20.397084 20600 raft_consensus.cc:729] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d0b0a0cbf047c18117987fa6a90f44, State: Initialized, Role: FOLLOWER
I20260924 07:15:20.397504 20600 consensus_queue.cc:260] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.397745 20599 raft_consensus.cc:348] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.398047 20599 raft_consensus.cc:374] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:20.398128 20599 raft_consensus.cc:729] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283a3810524845fba67f609e69f88e92, State: Initialized, Role: FOLLOWER
I20260924 07:15:20.398182 20581 heartbeater.cc:503] Master 127.0.98.62:37971 was elected leader, sending a full tablet report...
I20260924 07:15:20.398528 20600 ts_tablet_manager.cc:1428] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 07:15:20.398663 20599 consensus_queue.cc:260] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.399497 20319 heartbeater.cc:503] Master 127.0.98.62:37971 was elected leader, sending a full tablet report...
I20260924 07:15:20.399789 20599 ts_tablet_manager.cc:1428] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20260924 07:15:20.418649 20320 tablet.cc:2369] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:20.440961 20606 raft_consensus.cc:482] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:15:20.441190 20606 raft_consensus.cc:504] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.442365 20606 leader_election.cc:288] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d0b0a0cbf047c18117987fa6a90f44 (127.0.98.3:36987), e38b086fec7c4571b0f0c55de5df1b56 (127.0.98.2:41821)
I20260924 07:15:20.446276 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.442630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36987 (local address 127.0.98.1:54077)
0924 07:15:20.442990 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.443002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.443116 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.443349 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.443351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.443366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:20.443513 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.443518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.444295 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.444298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.445212 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.445222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.445833 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.445836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.445958 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.445979 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.446042 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.446112 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260924 07:15:20.446255 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.442797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54077 (local address 127.0.98.3:36987)
0924 07:15:20.442936 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.442941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.443080 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:15:20.443223 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.443225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.443266 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.443327 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:20.443609 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.444192 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.445333 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.445726 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.445825 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.445888 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.446066 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.446139 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:15:20.446762 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.442665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41821 (local address 127.0.98.1:42291)
0924 07:15:20.443074 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.443085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:20.443188 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:20.443644 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:20.443647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:20.443664 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:15:20.443803 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.443810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.444762 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.444766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.445626 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:20.445635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.446321 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.446324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:20.446416 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.446437 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:15:20.446549 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 07:15:20.446622 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":20,"thread_start_us":111,"threads_started":1}
I20260924 07:15:20.446787 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.442840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42291 (local address 127.0.98.2:41821)
0924 07:15:20.442962 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.442967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.443172 (+ 205us) server_negotiation.cc:407] Connection header received
0924 07:15:20.443288 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.443291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.443334 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.443419 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:15:20.443914 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.444660 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.445734 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.446204 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.446294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.446351 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.446590 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.446672 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:20.447154 20535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346e945f65054ff69a107ff4c429d8c8" candidate_uuid: "283a3810524845fba67f609e69f88e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d0b0a0cbf047c18117987fa6a90f44" is_pre_election: true
I20260924 07:15:20.447556 20535 raft_consensus.cc:2454] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283a3810524845fba67f609e69f88e92 in term 0.
I20260924 07:15:20.447525 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346e945f65054ff69a107ff4c429d8c8" candidate_uuid: "283a3810524845fba67f609e69f88e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38b086fec7c4571b0f0c55de5df1b56" is_pre_election: true
I20260924 07:15:20.447786 20404 raft_consensus.cc:2454] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 283a3810524845fba67f609e69f88e92 in term 0.
I20260924 07:15:20.448139 20206 leader_election.cc:302] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d0b0a0cbf047c18117987fa6a90f44, 283a3810524845fba67f609e69f88e92; no voters:
I20260924 07:15:20.448421 20606 raft_consensus.cc:2781] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:15:20.448501 20606 raft_consensus.cc:482] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:15:20.448552 20606 raft_consensus.cc:3037] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:20.449594 20606 raft_consensus.cc:504] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.449949 20606 leader_election.cc:288] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [CANDIDATE]: Term 1 election: Requested vote from peers 16d0b0a0cbf047c18117987fa6a90f44 (127.0.98.3:36987), e38b086fec7c4571b0f0c55de5df1b56 (127.0.98.2:41821)
I20260924 07:15:20.450326 20535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346e945f65054ff69a107ff4c429d8c8" candidate_uuid: "283a3810524845fba67f609e69f88e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d0b0a0cbf047c18117987fa6a90f44"
I20260924 07:15:20.450321 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "346e945f65054ff69a107ff4c429d8c8" candidate_uuid: "283a3810524845fba67f609e69f88e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e38b086fec7c4571b0f0c55de5df1b56"
I20260924 07:15:20.450455 20404 raft_consensus.cc:3037] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:20.450450 20535 raft_consensus.cc:3037] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:20.451555 20404 raft_consensus.cc:2454] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283a3810524845fba67f609e69f88e92 in term 1.
I20260924 07:15:20.451643 20535 raft_consensus.cc:2454] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 283a3810524845fba67f609e69f88e92 in term 1.
I20260924 07:15:20.451865 20208 leader_election.cc:302] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283a3810524845fba67f609e69f88e92, e38b086fec7c4571b0f0c55de5df1b56; no voters:
I20260924 07:15:20.452049 20606 raft_consensus.cc:2781] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:20.452262 20606 raft_consensus.cc:686] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [term 1 LEADER]: Becoming Leader. State: Replica: 283a3810524845fba67f609e69f88e92, State: Running, Role: LEADER
I20260924 07:15:20.452770 20606 consensus_queue.cc:237] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } }
I20260924 07:15:20.456491 20131 catalog_manager.cc:5495] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 283a3810524845fba67f609e69f88e92 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "283a3810524845fba67f609e69f88e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "283a3810524845fba67f609e69f88e92" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35399 } health_report { overall_health: HEALTHY } } }
I20260924 07:15:20.471833 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.468235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.0.98.1:35399)
0924 07:15:20.468363 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:20.468367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:20.468399 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:15:20.468472 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:20.468475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:20.468531 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:20.468636 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:20.468832 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.469319 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.470227 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:20.470436 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:20.470507 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:20.470818 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:20.471253 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:20.471338 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:20.471615 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 07:15:20.471681 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:20.478461 20535 raft_consensus.cc:1261] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 1 FOLLOWER]: Refusing update from remote peer 283a3810524845fba67f609e69f88e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:20.478475 20404 raft_consensus.cc:1261] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 1 FOLLOWER]: Refusing update from remote peer 283a3810524845fba67f609e69f88e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:20.479180 20609 consensus_queue.cc:1035] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d0b0a0cbf047c18117987fa6a90f44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:20.479396 20606 consensus_queue.cc:1035] T 346e945f65054ff69a107ff4c429d8c8 P 283a3810524845fba67f609e69f88e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38b086fec7c4571b0f0c55de5df1b56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:20.487954 20614 mvcc.cc:205] Tried to move back new op lower bound from 7332798957471559680 to 7332798957377470464. Current Snapshot: MvccSnapshot[applied={T|T < 7332798957471559680}]
I20260924 07:15:20.496604 20130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:20.496850 20130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:15:20.499493 20130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4081bdbedc674b308a3a5fee9e0abf63: Sending DeleteTablet for 3 replicas of tablet 346e945f65054ff69a107ff4c429d8c8
I20260924 07:15:20.500463 20384 tablet_service.cc:1514] Processing DeleteTablet for tablet 346e945f65054ff69a107ff4c429d8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:20 UTC) from {username='slave'} at 127.0.0.1:54626
I20260924 07:15:20.500602 20253 tablet_service.cc:1514] Processing DeleteTablet for tablet 346e945f65054ff69a107ff4c429d8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:20 UTC) from {username='slave'} at 127.0.0.1:37930
I20260924 07:15:20.500602 20515 tablet_service.cc:1514] Processing DeleteTablet for tablet 346e945f65054ff69a107ff4c429d8c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:15:20 UTC) from {username='slave'} at 127.0.0.1:58354
I20260924 07:15:20.501186 20623 tablet_replica.cc:330] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: stopping tablet replica
I20260924 07:15:20.501412 20621 tablet_replica.cc:330] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: stopping tablet replica
I20260924 07:15:20.501629 20623 raft_consensus.cc:2229] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:20.501781 20621 raft_consensus.cc:2229] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:15:20.502019 20623 raft_consensus.cc:2258] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:20.502090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20191
I20260924 07:15:20.502974 20621 raft_consensus.cc:2258] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:15:20.504323 20623 ts_tablet_manager.cc:1905] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:20.504400 20621 ts_tablet_manager.cc:1905] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:15:20.507202 20623 ts_tablet_manager.cc:1918] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:20.507202 20621 ts_tablet_manager.cc:1918] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:15:20.507321 20621 log.cc:1196] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-1/wal/wals/346e945f65054ff69a107ff4c429d8c8
I20260924 07:15:20.507321 20623 log.cc:1196] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Deleting WAL directory at /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233866113376-392-0/minicluster-data/ts-2/wal/wals/346e945f65054ff69a107ff4c429d8c8
I20260924 07:15:20.507694 20621 ts_tablet_manager.cc:1939] T 346e945f65054ff69a107ff4c429d8c8 P e38b086fec7c4571b0f0c55de5df1b56: Deleting consensus metadata
I20260924 07:15:20.507694 20623 ts_tablet_manager.cc:1939] T 346e945f65054ff69a107ff4c429d8c8 P 16d0b0a0cbf047c18117987fa6a90f44: Deleting consensus metadata
W20260924 07:15:20.508899 20116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35399 (error 108)
I20260924 07:15:20.509107 20117 catalog_manager.cc:4841] TS e38b086fec7c4571b0f0c55de5df1b56 (127.0.98.2:41821): tablet 346e945f65054ff69a107ff4c429d8c8 (table test-table [id=f11b3bbef26d4658b84b2e168559d229]) successfully deleted
I20260924 07:15:20.509331 20115 catalog_manager.cc:4841] TS 16d0b0a0cbf047c18117987fa6a90f44 (127.0.98.3:36987): tablet 346e945f65054ff69a107ff4c429d8c8 (table test-table [id=f11b3bbef26d4658b84b2e168559d229]) successfully deleted
I20260924 07:15:20.509761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20322
I20260924 07:15:20.510028 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:20.509718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35399 (local address 127.0.0.1:37960)
0924 07:15:20.509864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:20.509939 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 07:15:20.510547 20116 catalog_manager.cc:4584] TS 283a3810524845fba67f609e69f88e92 (127.0.98.1:35399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 346e945f65054ff69a107ff4c429d8c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35399: connect: Connection refused (error 111)
I20260924 07:15:20.515427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20453
I20260924 07:15:20.520298 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20100
2026-09-24T07:15:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1080 ms)
[----------] 6 tests from EncryptionPolicyTest (10145 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:20 dist-test-slave-cnv3 krb5kdc[20627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:20 dist-test-slave-cnv3 krb5kdc[20627](info): set up 2 sockets
Sep 24 07:15:20 dist-test-slave-cnv3 krb5kdc[20627](info): commencing operation
krb5kdc: starting...
W20260924 07:15:21.483691 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:21.608434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:21.717878 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:21.718168 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:21.718211 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:21.722337 20640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:21.722414 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:21.722432 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:21.722448 20640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:21.722462 20640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:21.728044 20640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:21.729215 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:21.730284 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:21.738787 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:21.738894 20648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:21.739136 20640 server_base.cc:1034] running on GCE node
W20260924 07:15:21.739279 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:21.739892 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:21.741110 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:21.742290 20640 hybrid_clock.cc:648] HybridClock initialized: now 1790234121742267 us; error 41 us; skew 500 ppm
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:21.744972 20640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:21.745916 20640 webserver.cc:460] Webserver started at http://127.0.0.1:34037/ using document root <none> and password file <none>
I20260924 07:15:21.746433 20640 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:21.746486 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:21.746650 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:21.748695 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "5de6e86739d64266a20afa653865a0c4"
format_stamp: "Formatted at 2026-09-24 07:15:21 on dist-test-slave-cnv3"
server_key: "73421ea493701af1aeaa15ca5c077779"
server_key_iv: "c806badfca2785ee0dcce516d005f47e"
server_key_version: "encryptionkey@0"
I20260924 07:15:21.749251 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "5de6e86739d64266a20afa653865a0c4"
format_stamp: "Formatted at 2026-09-24 07:15:21 on dist-test-slave-cnv3"
server_key: "73421ea493701af1aeaa15ca5c077779"
server_key_iv: "c806badfca2785ee0dcce516d005f47e"
server_key_version: "encryptionkey@0"
I20260924 07:15:21.753067 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:21.755497 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:21.756798 20640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:21.756904 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "5de6e86739d64266a20afa653865a0c4"
format_stamp: "Formatted at 2026-09-24 07:15:21 on dist-test-slave-cnv3"
server_key: "73421ea493701af1aeaa15ca5c077779"
server_key_iv: "c806badfca2785ee0dcce516d005f47e"
server_key_version: "encryptionkey@0"
I20260924 07:15:21.757005 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:21.770467 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:21.774025 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:21.774192 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:21.781132 20640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46537
I20260924 07:15:21.781190 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46537 every 8 connection(s)
I20260924 07:15:21.782104 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:21.784909 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:21.785069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20640
I20260924 07:15:21.785184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:21.785422 392 external_mini_cluster.cc:1428] Setting key 5968348eb95a30db84803fe0762d5d53
I20260924 07:15:21.792037 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4: Bootstrap starting.
I20260924 07:15:21.794447 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4: Neither blocks nor log segments found. Creating new log.
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:15:21.795212 20708 log.cc:824] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:21.797523 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4: No bootstrap required, opened a new log
I20260924 07:15:21.800379 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:21.786863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55531 (local address 127.0.0.1:46537)
0924 07:15:21.787302 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:21.787321 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:21.787363 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:15:21.788173 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:21.788198 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:21.788648 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:21.789013 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:21.789709 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:21.791222 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:21.792107 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:21.792772 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:21.792918 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:21.795276 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:21.795282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:21.795299 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:21.795332 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:21.797243 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:21.797704 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:21.797710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:21.797717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:21.797858 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:21.798131 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:21.798134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:21.798135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:21.798653 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:21.798857 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:21.799079 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 07:15:21.799337 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260924 07:15:21.802115 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } }
I20260924 07:15:21.802414 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:21.802502 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de6e86739d64266a20afa653865a0c4, State: Initialized, Role: FOLLOWER
I20260924 07:15:21.802964 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } }
I20260924 07:15:21.803114 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:21.803179 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:21.803273 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:21.803442 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:21.804308 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } }
I20260924 07:15:21.804662 20708 leader_election.cc:302] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5de6e86739d64266a20afa653865a0c4; no voters:
I20260924 07:15:21.804967 20708 leader_election.cc:288] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:21.805095 20717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:21.805331 20717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5de6e86739d64266a20afa653865a0c4, State: Running, Role: LEADER
I20260924 07:15:21.805716 20717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } }
I20260924 07:15:21.806294 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:21.807013 20719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5de6e86739d64266a20afa653865a0c4. Latest consensus state: current_term: 1 leader_uuid: "5de6e86739d64266a20afa653865a0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } } }
I20260924 07:15:21.807142 20719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:21.807829 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5de6e86739d64266a20afa653865a0c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de6e86739d64266a20afa653865a0c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46537 } } }
I20260924 07:15:21.808002 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:21.808821 20726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:21.812084 20726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:21.814988 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:21.805716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55551 (local address 127.0.0.1:46537)
0924 07:15:21.805872 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:21.805876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:21.805890 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:15:21.805928 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:21.805930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:21.805976 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:21.806105 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:21.806951 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:21.807777 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:21.808929 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:21.809217 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:21.809312 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:21.810928 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:21.810932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:21.810935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:21.810960 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:21.812631 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:21.813206 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:21.813210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:21.813211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:21.813298 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:21.813697 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:21.813700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:21.813703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:21.813939 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:21.814082 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:21.814713 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 07:15:21.814828 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 07:15:21.818316 20726 catalog_manager.cc:1347] Generated new cluster ID: 55ddaa687d1141289b7076b5629e8408
I20260924 07:15:21.818392 20726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:21.838279 20726 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:21.839699 20726 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:21.852838 20726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5de6e86739d64266a20afa653865a0c4: Generated new TSK 0
I20260924 07:15:21.853857 20726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:21.861889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20640
W20260924 07:15:21.867033 20625 connection.cc:535] client connection to 127.0.0.1:46537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46537): Connection reset by peer (error 104)
2026-09-24T07:15:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20739](info): set up 2 sockets
Sep 24 07:15:21 dist-test-slave-cnv3 krb5kdc[20739](info): commencing operation
krb5kdc: starting...
W20260924 07:15:22.823525 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:22 dist-test-slave-cnv3 krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:22.946112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:23.055110 20752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:23.055447 20752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:23.055513 20752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:23.059638 20752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:23.059713 20752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:23.059741 20752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:23.059762 20752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:23.059783 20752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:23.065368 20752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:23.066497 20752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:23.067533 20752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:23.076268 20760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:23.076264 20758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:23.076264 20757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:23.076668 20752 server_base.cc:1034] running on GCE node
I20260924 07:15:23.077224 20752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:23.078504 20752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:23.079687 20752 hybrid_clock.cc:648] HybridClock initialized: now 1790234123079650 us; error 53 us; skew 500 ppm
Sep 24 07:15:23 dist-test-slave-cnv3 krb5kdc[20739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:23.082651 20752 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:23.083638 20752 webserver.cc:460] Webserver started at http://127.0.0.1:43169/ using document root <none> and password file <none>
I20260924 07:15:23.084210 20752 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:23.084280 20752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:23.084476 20752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:23.086587 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "48ab57cc026f4761a58ccd316e9529f6"
format_stamp: "Formatted at 2026-09-24 07:15:23 on dist-test-slave-cnv3"
server_key: "8eec2a2d58a535a92d86e2c617b3cf02"
server_key_iv: "7e649c4556fda1be9dc2188cc64f5b5e"
server_key_version: "encryptionkey@0"
I20260924 07:15:23.087175 20752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "48ab57cc026f4761a58ccd316e9529f6"
format_stamp: "Formatted at 2026-09-24 07:15:23 on dist-test-slave-cnv3"
server_key: "8eec2a2d58a535a92d86e2c617b3cf02"
server_key_iv: "7e649c4556fda1be9dc2188cc64f5b5e"
server_key_version: "encryptionkey@0"
I20260924 07:15:23.091060 20752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260924 07:15:23.093670 20767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:23.094919 20752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:15:23.095044 20752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "48ab57cc026f4761a58ccd316e9529f6"
format_stamp: "Formatted at 2026-09-24 07:15:23 on dist-test-slave-cnv3"
server_key: "8eec2a2d58a535a92d86e2c617b3cf02"
server_key_iv: "7e649c4556fda1be9dc2188cc64f5b5e"
server_key_version: "encryptionkey@0"
I20260924 07:15:23.095155 20752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:23.112531 20752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:23.115615 20752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:23.115803 20752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:23.122594 20752 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40851
I20260924 07:15:23.122646 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40851 every 8 connection(s)
I20260924 07:15:23.123610 20752 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:23.126348 20820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:23.131911 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6: Bootstrap starting.
I20260924 07:15:23.132731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20752
I20260924 07:15:23.132859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:23.133150 392 external_mini_cluster.cc:1428] Setting key a4c60007728f1f8307acc8ec3d99e528
I20260924 07:15:23.134780 20820 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:23.135555 20820 log.cc:824] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:23.137434 20820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6: No bootstrap required, opened a new log
I20260924 07:15:23.141165 20820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260924 07:15:23.141443 20820 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:23.141492 20820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ab57cc026f4761a58ccd316e9529f6, State: Initialized, Role: FOLLOWER
I20260924 07:15:23.141947 20820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260924 07:15:23.142068 20820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:23.142119 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:23.134268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55389 (local address 127.0.0.1:40851)
0924 07:15:23.134709 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:23.134724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:23.134759 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:23.135501 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:23.135521 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:23.135913 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:23.136227 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:23.136629 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:23.137660 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:23.138541 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:23.139042 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:23.139198 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:23.140148 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:23.140154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:23.140169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:23.140606 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:23.140664 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:23.140690 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:23.140815 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:23.141004 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 07:15:23.141149 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260924 07:15:23.142130 20820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:23.142278 20820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:23.143289 20820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260924 07:15:23.143687 20820 leader_election.cc:302] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48ab57cc026f4761a58ccd316e9529f6; no voters:
I20260924 07:15:23.144018 20820 leader_election.cc:288] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:23.144120 20825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:23.144356 20825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [term 1 LEADER]: Becoming Leader. State: Replica: 48ab57cc026f4761a58ccd316e9529f6, State: Running, Role: LEADER
I20260924 07:15:23.145037 20825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } }
I20260924 07:15:23.145191 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:23.145421 20820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:23.146642 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48ab57cc026f4761a58ccd316e9529f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } } }
I20260924 07:15:23.146752 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:23.147044 20827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48ab57cc026f4761a58ccd316e9529f6. Latest consensus state: current_term: 1 leader_uuid: "48ab57cc026f4761a58ccd316e9529f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48ab57cc026f4761a58ccd316e9529f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40851 } } }
I20260924 07:15:23.147167 20827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:23.147466 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:23.151046 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:23.152096 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:23.148078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39377 (local address 127.0.0.1:40851)
0924 07:15:23.148196 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:23.148200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:23.148546 (+ 346us) server_negotiation.cc:407] Connection header received
0924 07:15:23.148663 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:23.148666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:23.148709 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:23.148771 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:23.149024 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:23.149543 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:23.150476 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:23.150697 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:23.150780 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:23.151630 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:23.151634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:23.151637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:23.151650 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:23.151674 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:23.151685 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:23.151764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:23.151919 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:15:23.151953 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:15:23.157618 20839 catalog_manager.cc:1347] Generated new cluster ID: 3214b921a77e4fd0ab3d16be5627c96f
I20260924 07:15:23.157706 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:23.168835 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:23.169806 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:23.173774 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48ab57cc026f4761a58ccd316e9529f6: Generated new TSK 0
I20260924 07:15:23.174664 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:23.179091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20752
W20260924 07:15:23.183871 20737 connection.cc:535] client connection to 127.0.0.1:40851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40851): Connection reset by peer (error 104)
2026-09-24T07:15:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:23 dist-test-slave-cnv3 krb5kdc[20851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:23 dist-test-slave-cnv3 krb5kdc[20851](info): set up 2 sockets
Sep 24 07:15:23 dist-test-slave-cnv3 krb5kdc[20851](info): commencing operation
krb5kdc: starting...
W20260924 07:15:24.135368 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:24 dist-test-slave-cnv3 krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:24.258239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:24.369603 20864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:24.369900 20864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:24.369961 20864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:24.374161 20864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:24.374238 20864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:24.374265 20864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:24.374286 20864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:24.374307 20864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:24.379922 20864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:24.381083 20864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:24.382114 20864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:24.390663 20870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:24.390831 20864 server_base.cc:1034] running on GCE node
W20260924 07:15:24.390673 20872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:24.390669 20869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:24.391630 20864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:24.392788 20864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:24.393965 20864 hybrid_clock.cc:648] HybridClock initialized: now 1790234124393941 us; error 38 us; skew 500 ppm
Sep 24 07:15:24 dist-test-slave-cnv3 krb5kdc[20851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:24.396759 20864 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:24.397660 20864 webserver.cc:460] Webserver started at http://127.0.0.1:37649/ using document root <none> and password file <none>
I20260924 07:15:24.398201 20864 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:24.398268 20864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:24.398454 20864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:24.400583 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb"
format_stamp: "Formatted at 2026-09-24 07:15:24 on dist-test-slave-cnv3"
server_key: "a74119c190fcc3c3b18943626aa22fa7"
server_key_iv: "197507eb5e49c245dd5823ed498fd05e"
server_key_version: "encryptionkey@0"
I20260924 07:15:24.401110 20864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb"
format_stamp: "Formatted at 2026-09-24 07:15:24 on dist-test-slave-cnv3"
server_key: "a74119c190fcc3c3b18943626aa22fa7"
server_key_iv: "197507eb5e49c245dd5823ed498fd05e"
server_key_version: "encryptionkey@0"
I20260924 07:15:24.404910 20864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:24.407503 20879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:24.408957 20864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:24.409116 20864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb"
format_stamp: "Formatted at 2026-09-24 07:15:24 on dist-test-slave-cnv3"
server_key: "a74119c190fcc3c3b18943626aa22fa7"
server_key_iv: "197507eb5e49c245dd5823ed498fd05e"
server_key_version: "encryptionkey@0"
I20260924 07:15:24.409231 20864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:24.414062 20864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:24.417253 20864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:24.417433 20864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:24.424633 20864 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34583
I20260924 07:15:24.424659 20931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34583 every 8 connection(s)
I20260924 07:15:24.425612 20864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:24.428447 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:24.434103 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb: Bootstrap starting.
I20260924 07:15:24.434545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20864
I20260924 07:15:24.434674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:24.434968 392 external_mini_cluster.cc:1428] Setting key 8d6b33ebbad6e9e99ba369484088058d
I20260924 07:15:24.436831 20932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:24.437549 20932 log.cc:824] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:24.439791 20932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb: No bootstrap required, opened a new log
I20260924 07:15:24.440440 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:24.436405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59249 (local address 127.0.0.1:34583)
0924 07:15:24.436880 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:24.436894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:24.436932 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:15:24.437665 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:24.437689 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:24.438046 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:24.438421 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:24.439201 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:24.439205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:24.439217 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:24.439227 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:24.439278 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:24.439296 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:24.439406 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:24.439535 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 07:15:24.439701 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":135,"threads_started":1}
I20260924 07:15:24.443307 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:24.443883 20932 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260924 07:15:24.444168 20932 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:24.444224 20932 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5000823fbe4b4db7a6b4de5b2f63cb, State: Initialized, Role: FOLLOWER
I20260924 07:15:24.444710 20932 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260924 07:15:24.444885 20932 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:24.444957 20932 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:24.445051 20932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:24.446079 20932 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260924 07:15:24.446385 20932 leader_election.cc:302] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5000823fbe4b4db7a6b4de5b2f63cb; no voters:
I20260924 07:15:24.446662 20932 leader_election.cc:288] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:24.446761 20942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:24.446979 20942 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [term 1 LEADER]: Becoming Leader. State: Replica: ea5000823fbe4b4db7a6b4de5b2f63cb, State: Running, Role: LEADER
I20260924 07:15:24.447235 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:24.445575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39849 (local address 127.0.0.1:34583)
0924 07:15:24.445694 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:24.445698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:24.445972 (+ 274us) server_negotiation.cc:407] Connection header received
0924 07:15:24.446016 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:24.446019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:24.446053 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:24.446118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:24.446791 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:24.446794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:24.446797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:24.446809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:24.446837 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:24.446850 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:24.446907 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:24.447085 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:15:24.447127 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:15:24.447338 20942 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } }
I20260924 07:15:24.447932 20932 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:24.449105 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260924 07:15:24.449206 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:24.449568 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:24.449523 20943 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5000823fbe4b4db7a6b4de5b2f63cb. Latest consensus state: current_term: 1 leader_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5000823fbe4b4db7a6b4de5b2f63cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34583 } } }
I20260924 07:15:24.450632 20943 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:24.452773 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:24.458017 20952 catalog_manager.cc:1347] Generated new cluster ID: c563408406094f48b7ca2096cc9f32cf
I20260924 07:15:24.458101 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:24.482822 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:24.483184 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:24.490703 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5000823fbe4b4db7a6b4de5b2f63cb: Generated new TSK 0
I20260924 07:15:24.491449 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:24.500195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20864
2026-09-24T07:15:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:24 dist-test-slave-cnv3 krb5kdc[20963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:24 dist-test-slave-cnv3 krb5kdc[20963](info): set up 2 sockets
Sep 24 07:15:24 dist-test-slave-cnv3 krb5kdc[20963](info): commencing operation
krb5kdc: starting...
W20260924 07:15:25.459343 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:25.578732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:25.689939 20976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:25.690243 20976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:25.690303 20976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:25.694532 20976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:25.694609 20976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:25.694635 20976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:25.694658 20976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:25.694686 20976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:25.700346 20976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:25.701499 20976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:25.702546 20976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:25.711741 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:25.711746 20981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:25.711998 20976 server_base.cc:1034] running on GCE node
W20260924 07:15:25.711799 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:25.712710 20976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:25.713915 20976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:25.715116 20976 hybrid_clock.cc:648] HybridClock initialized: now 1790234125715093 us; error 41 us; skew 500 ppm
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[20963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:25.721818 20976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:25.722839 20976 webserver.cc:460] Webserver started at http://127.0.0.1:35537/ using document root <none> and password file <none>
I20260924 07:15:25.723490 20976 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:25.723582 20976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:25.723805 20976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:25.725971 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "85848d2de466473eb6b99231c7941891"
format_stamp: "Formatted at 2026-09-24 07:15:25 on dist-test-slave-cnv3"
server_key: "ccba040ed5074cd530a31237f8d3b0e3"
server_key_iv: "8f517aae644f9394d7938070aa4a0881"
server_key_version: "encryptionkey@0"
I20260924 07:15:25.726519 20976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "85848d2de466473eb6b99231c7941891"
format_stamp: "Formatted at 2026-09-24 07:15:25 on dist-test-slave-cnv3"
server_key: "ccba040ed5074cd530a31237f8d3b0e3"
server_key_iv: "8f517aae644f9394d7938070aa4a0881"
server_key_version: "encryptionkey@0"
I20260924 07:15:25.730446 20976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:25.733129 20991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:25.734467 20976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:25.734611 20976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "85848d2de466473eb6b99231c7941891"
format_stamp: "Formatted at 2026-09-24 07:15:25 on dist-test-slave-cnv3"
server_key: "ccba040ed5074cd530a31237f8d3b0e3"
server_key_iv: "8f517aae644f9394d7938070aa4a0881"
server_key_version: "encryptionkey@0"
I20260924 07:15:25.734735 20976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:25.743803 20976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:25.747008 20976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:25.747195 20976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:25.753961 20976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44293
I20260924 07:15:25.753980 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44293 every 8 connection(s)
I20260924 07:15:25.754941 20976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:25.755125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 20976
I20260924 07:15:25.755237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:25.755559 392 external_mini_cluster.cc:1428] Setting key e6902e24ff2d66ff1a89381dd2f99ac9
I20260924 07:15:25.757989 21044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[20963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:15:25.764994 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891: Bootstrap starting.
I20260924 07:15:25.767938 21044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:25.768712 21044 log.cc:824] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:25.770061 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:25.756942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40427 (local address 127.0.0.1:44293)
0924 07:15:25.757341 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:25.757359 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:25.757397 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:15:25.758023 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:25.758039 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:25.758424 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:25.758698 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:25.759604 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:25.760627 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:25.761941 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:25.762455 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:25.762576 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:25.765283 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:25.765290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:25.765307 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:25.765342 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:25.767180 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:25.767637 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:25.767643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:25.767649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:25.767811 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:25.768083 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:25.768086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:25.768087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:25.768525 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:25.768705 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:25.769000 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:15:25.769335 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260924 07:15:25.770666 21044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891: No bootstrap required, opened a new log
I20260924 07:15:25.773026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:25.775170 21044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } }
I20260924 07:15:25.775485 21044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:25.775545 21044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85848d2de466473eb6b99231c7941891, State: Initialized, Role: FOLLOWER
I20260924 07:15:25.775988 21044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } }
I20260924 07:15:25.776129 21044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:25.776191 21044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:25.776286 21044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:25.777240 21044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } }
I20260924 07:15:25.777575 21044 leader_election.cc:302] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85848d2de466473eb6b99231c7941891; no voters:
I20260924 07:15:25.777868 21044 leader_election.cc:288] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:25.777980 21054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:25.778208 21054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [term 1 LEADER]: Becoming Leader. State: Replica: 85848d2de466473eb6b99231c7941891, State: Running, Role: LEADER
I20260924 07:15:25.778617 21054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } }
I20260924 07:15:25.779131 21044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:25.779984 21055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85848d2de466473eb6b99231c7941891. Latest consensus state: current_term: 1 leader_uuid: "85848d2de466473eb6b99231c7941891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } } }
I20260924 07:15:25.780611 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85848d2de466473eb6b99231c7941891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85848d2de466473eb6b99231c7941891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44293 } } }
I20260924 07:15:25.780738 21055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:25.780844 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:25.781284 21068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:25.784309 21047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:25.775240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57459 (local address 127.0.0.1:44293)
0924 07:15:25.775397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:25.775402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:25.775637 (+ 235us) server_negotiation.cc:407] Connection header received
0924 07:15:25.775732 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:25.775735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:25.775780 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:25.775843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:25.776593 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:25.777109 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:25.778091 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:25.778419 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:25.778496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:25.780361 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:25.780365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:25.780368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:25.780400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:25.782020 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:25.782534 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:25.782538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:25.782540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:25.782631 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:25.782950 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:25.782953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:25.782955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:25.783198 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:25.783321 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:25.784003 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 07:15:25.784144 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:15:25.785197 21068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:25.791203 21068 catalog_manager.cc:1347] Generated new cluster ID: 220811d8888b412abb8e32ff14c522d8
I20260924 07:15:25.791303 21068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:25.805095 21068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:25.806454 21068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:25.822853 21068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85848d2de466473eb6b99231c7941891: Generated new TSK 0
I20260924 07:15:25.823837 21068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:25.836052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 20976
2026-09-24T07:15:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[21075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[21075](info): set up 2 sockets
Sep 24 07:15:25 dist-test-slave-cnv3 krb5kdc[21075](info): commencing operation
krb5kdc: starting...
W20260924 07:15:26.807355 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:26 dist-test-slave-cnv3 krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:26.932667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:27.045609 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:27.045953 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:27.046017 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:27.050287 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:27.050379 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:27.050405 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:27.050436 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:27.050457 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:27.056300 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:27.057566 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:27.058629 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:27.067698 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:27.067783 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:27.067812 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:27.067896 21088 server_base.cc:1034] running on GCE node
I20260924 07:15:27.068640 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:27.069852 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:27.071053 21088 hybrid_clock.cc:648] HybridClock initialized: now 1790234127071033 us; error 43 us; skew 500 ppm
Sep 24 07:15:27 dist-test-slave-cnv3 krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:27.074291 21088 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:27.075289 21088 webserver.cc:460] Webserver started at http://127.0.0.1:40405/ using document root <none> and password file <none>
I20260924 07:15:27.075888 21088 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:27.075958 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:27.076151 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:27.078239 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "67fe872f2c34414e860b91a9c126cbe7"
format_stamp: "Formatted at 2026-09-24 07:15:27 on dist-test-slave-cnv3"
server_key: "7b56b85b82e2bbe05473be61364d5b2a"
server_key_iv: "020f8ed639b5f628f9624c09ca587315"
server_key_version: "encryptionkey@0"
I20260924 07:15:27.078779 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "67fe872f2c34414e860b91a9c126cbe7"
format_stamp: "Formatted at 2026-09-24 07:15:27 on dist-test-slave-cnv3"
server_key: "7b56b85b82e2bbe05473be61364d5b2a"
server_key_iv: "020f8ed639b5f628f9624c09ca587315"
server_key_version: "encryptionkey@0"
I20260924 07:15:27.082659 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:27.085181 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:27.086746 21088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:15:27.086882 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "67fe872f2c34414e860b91a9c126cbe7"
format_stamp: "Formatted at 2026-09-24 07:15:27 on dist-test-slave-cnv3"
server_key: "7b56b85b82e2bbe05473be61364d5b2a"
server_key_iv: "020f8ed639b5f628f9624c09ca587315"
server_key_version: "encryptionkey@0"
I20260924 07:15:27.087001 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:27.114746 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:27.118209 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:27.118458 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:27.125520 21088 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40345
I20260924 07:15:27.125537 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40345 every 8 connection(s)
I20260924 07:15:27.126504 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:27.129364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21088
I20260924 07:15:27.129510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:27.129568 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:27.129801 392 external_mini_cluster.cc:1428] Setting key 517c9271a8c891ca7e59944b1c677100
I20260924 07:15:27.136314 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: Bootstrap starting.
I20260924 07:15:27.138742 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:27.139559 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:27.131299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34621 (local address 127.0.0.1:40345)
0924 07:15:27.131833 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:27.131849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:27.131889 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:15:27.132846 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:27.132868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:27.133232 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:27.133554 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:27.134087 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:27.135191 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:27.136154 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:27.136672 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:27.136826 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:27.137742 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:27.137748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:27.137761 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:27.138154 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:27.138220 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:27.138246 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:27.138330 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:27.138473 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:15:27.138681 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":146,"threads_started":1}
I20260924 07:15:27.139555 21156 log.cc:824] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:27.141748 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: No bootstrap required, opened a new log
I20260924 07:15:27.142499 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:27.146085 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } }
I20260924 07:15:27.146412 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:27.146472 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67fe872f2c34414e860b91a9c126cbe7, State: Initialized, Role: FOLLOWER
I20260924 07:15:27.146950 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } }
I20260924 07:15:27.147120 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:27.147190 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:27.147297 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:27.148528 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } }
I20260924 07:15:27.148793 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:27.144486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53905 (local address 127.0.0.1:40345)
0924 07:15:27.144598 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:27.144603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:27.144894 (+ 291us) server_negotiation.cc:407] Connection header received
0924 07:15:27.144975 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:27.144977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:27.145020 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:27.145090 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:27.145388 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:27.145920 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:27.147019 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:27.147414 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:27.147505 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:27.148336 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:27.148340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:27.148342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:27.148353 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:27.148380 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:27.148390 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:27.148455 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:27.148576 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:15:27.148628 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:15:27.148934 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67fe872f2c34414e860b91a9c126cbe7; no voters:
I20260924 07:15:27.149291 21156 leader_election.cc:288] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:27.149374 21166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:27.149551 21166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [term 1 LEADER]: Becoming Leader. State: Replica: 67fe872f2c34414e860b91a9c126cbe7, State: Running, Role: LEADER
I20260924 07:15:27.149930 21166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } }
I20260924 07:15:27.150682 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:27.151803 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67fe872f2c34414e860b91a9c126cbe7. Latest consensus state: current_term: 1 leader_uuid: "67fe872f2c34414e860b91a9c126cbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } } }
I20260924 07:15:27.151803 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67fe872f2c34414e860b91a9c126cbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67fe872f2c34414e860b91a9c126cbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40345 } } }
I20260924 07:15:27.151983 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:27.151981 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7 [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:27.156052 21181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:27.156157 21181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:27.156257 21175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:27.157094 21175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:27.162989 21175 catalog_manager.cc:1347] Generated new cluster ID: fbb96b37c2a6409fb6c47af575c8b407
I20260924 07:15:27.163069 21175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:27.175446 21175 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:27.176800 21175 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:27.183219 21175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67fe872f2c34414e860b91a9c126cbe7: Generated new TSK 0
I20260924 07:15:27.184113 21175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:27.193976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21088
2026-09-24T07:15:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:27 dist-test-slave-cnv3 krb5kdc[21187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:27 dist-test-slave-cnv3 krb5kdc[21187](info): set up 2 sockets
Sep 24 07:15:27 dist-test-slave-cnv3 krb5kdc[21187](info): commencing operation
krb5kdc: starting...
W20260924 07:15:28.159793 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:28 dist-test-slave-cnv3 krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:28.288002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:28.402303 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:28.402611 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:28.402653 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:28.406847 21200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:28.406922 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:28.406942 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:28.406957 21200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:28.406972 21200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:28.412564 21200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:28.413677 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:28.414675 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:28.423251 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:28.423236 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:28.423240 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:28.423344 21200 server_base.cc:1034] running on GCE node
I20260924 07:15:28.424165 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:28.425288 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:28.426455 21200 hybrid_clock.cc:648] HybridClock initialized: now 1790234128426439 us; error 30 us; skew 500 ppm
Sep 24 07:15:28 dist-test-slave-cnv3 krb5kdc[21187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:28.429188 21200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:28.430155 21200 webserver.cc:460] Webserver started at http://127.0.0.1:43109/ using document root <none> and password file <none>
I20260924 07:15:28.430735 21200 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:28.430805 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:28.431011 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:28.433286 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "04c48b5b63014a3db40b7615accb7dce"
format_stamp: "Formatted at 2026-09-24 07:15:28 on dist-test-slave-cnv3"
server_key: "63a70c5386c318a7c1181f3918920fff"
server_key_iv: "15c8dc62b6d3a17d4341795f4bab5bcf"
server_key_version: "encryptionkey@0"
I20260924 07:15:28.433894 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "04c48b5b63014a3db40b7615accb7dce"
format_stamp: "Formatted at 2026-09-24 07:15:28 on dist-test-slave-cnv3"
server_key: "63a70c5386c318a7c1181f3918920fff"
server_key_iv: "15c8dc62b6d3a17d4341795f4bab5bcf"
server_key_version: "encryptionkey@0"
I20260924 07:15:28.438153 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:15:28.440853 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:28.442384 21200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:15:28.442531 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "04c48b5b63014a3db40b7615accb7dce"
format_stamp: "Formatted at 2026-09-24 07:15:28 on dist-test-slave-cnv3"
server_key: "63a70c5386c318a7c1181f3918920fff"
server_key_iv: "15c8dc62b6d3a17d4341795f4bab5bcf"
server_key_version: "encryptionkey@0"
I20260924 07:15:28.442659 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:28.447649 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:28.450735 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:28.450909 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:28.457803 21200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43735
I20260924 07:15:28.457818 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43735 every 8 connection(s)
I20260924 07:15:28.458761 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:28.461638 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:28.464432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21200
I20260924 07:15:28.464586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:28.464855 392 external_mini_cluster.cc:1428] Setting key 498d2679ace9328deb32351332b825d5
I20260924 07:15:28.467778 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce: Bootstrap starting.
I20260924 07:15:28.470052 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:28.466250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39977 (local address 127.0.0.1:43735)
0924 07:15:28.466670 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:28.466684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:28.466720 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:28.467324 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:28.467341 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:28.467683 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:28.468001 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:28.468773 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:28.468778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:28.468789 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:28.468799 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:28.468853 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:28.468872 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:28.469050 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:28.469109 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:15:28.469267 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260924 07:15:28.470244 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:28.471158 21268 log.cc:824] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:28.473133 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:28.473430 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce: No bootstrap required, opened a new log
I20260924 07:15:28.477201 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:28.475367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46859 (local address 127.0.0.1:43735)
0924 07:15:28.475532 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:28.475537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:28.475770 (+ 233us) server_negotiation.cc:407] Connection header received
0924 07:15:28.475931 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:28.475934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:28.475972 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:28.476046 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:28.476822 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:28.476825 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:28.476827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:28.476837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:28.476865 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:28.476875 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:28.476940 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:28.477024 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:15:28.477058 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:15:28.478009 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260924 07:15:28.478322 21268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:28.478386 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c48b5b63014a3db40b7615accb7dce, State: Initialized, Role: FOLLOWER
I20260924 07:15:28.478814 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260924 07:15:28.478940 21268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:28.478987 21268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:28.479059 21268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:28.480067 21268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260924 07:15:28.480387 21268 leader_election.cc:302] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04c48b5b63014a3db40b7615accb7dce; no voters:
I20260924 07:15:28.480759 21268 leader_election.cc:288] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:28.480840 21278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:28.481098 21278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [term 1 LEADER]: Becoming Leader. State: Replica: 04c48b5b63014a3db40b7615accb7dce, State: Running, Role: LEADER
I20260924 07:15:28.481482 21278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260924 07:15:28.482133 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:28.483083 21279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04c48b5b63014a3db40b7615accb7dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260924 07:15:28.483223 21279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:28.483505 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c48b5b63014a3db40b7615accb7dce. Latest consensus state: current_term: 1 leader_uuid: "04c48b5b63014a3db40b7615accb7dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04c48b5b63014a3db40b7615accb7dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260924 07:15:28.483596 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:28.483609 21287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:28.486397 21287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:28.492249 21287 catalog_manager.cc:1347] Generated new cluster ID: b1b535c5f286435a9a19777f2c17ca55
I20260924 07:15:28.492340 21287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:28.518414 21287 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:28.518795 21287 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:28.526686 21287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c48b5b63014a3db40b7615accb7dce: Generated new TSK 0
I20260924 07:15:28.527501 21287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:28.535564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21200
2026-09-24T07:15:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:28 dist-test-slave-cnv3 krb5kdc[21299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:28 dist-test-slave-cnv3 krb5kdc[21299](info): set up 2 sockets
Sep 24 07:15:28 dist-test-slave-cnv3 krb5kdc[21299](info): commencing operation
krb5kdc: starting...
W20260924 07:15:29.503348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:29.626772 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:29.739452 21312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:29.739748 21312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:29.739790 21312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:29.744027 21312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:29.744102 21312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:29.744122 21312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:29.744138 21312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:29.744154 21312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:29.749715 21312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:29.750818 21312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:29.752005 21312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:29.760848 21318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:29.760848 21320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:29.760879 21317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:29.761219 21312 server_base.cc:1034] running on GCE node
I20260924 07:15:29.761865 21312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:29.763051 21312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:29.764232 21312 hybrid_clock.cc:648] HybridClock initialized: now 1790234129764205 us; error 41 us; skew 500 ppm
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:29.767027 21312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:29.768008 21312 webserver.cc:460] Webserver started at http://127.0.0.1:44521/ using document root <none> and password file <none>
I20260924 07:15:29.768648 21312 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:29.768743 21312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:29.769017 21312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:29.771179 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "126dc3214d624fd2a4bc226a1fdd6d36"
format_stamp: "Formatted at 2026-09-24 07:15:29 on dist-test-slave-cnv3"
server_key: "585345fc320e9aea92dcb3e290f85c65"
server_key_iv: "f75de37ea8e4fa97cb7a9601e3df9af5"
server_key_version: "encryptionkey@0"
I20260924 07:15:29.771793 21312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "126dc3214d624fd2a4bc226a1fdd6d36"
format_stamp: "Formatted at 2026-09-24 07:15:29 on dist-test-slave-cnv3"
server_key: "585345fc320e9aea92dcb3e290f85c65"
server_key_iv: "f75de37ea8e4fa97cb7a9601e3df9af5"
server_key_version: "encryptionkey@0"
I20260924 07:15:29.775650 21312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:29.778156 21327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:29.779536 21312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:29.779663 21312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "126dc3214d624fd2a4bc226a1fdd6d36"
format_stamp: "Formatted at 2026-09-24 07:15:29 on dist-test-slave-cnv3"
server_key: "585345fc320e9aea92dcb3e290f85c65"
server_key_iv: "f75de37ea8e4fa97cb7a9601e3df9af5"
server_key_version: "encryptionkey@0"
I20260924 07:15:29.779783 21312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:29.812026 21312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:29.815057 21312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:29.815233 21312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:29.822321 21312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38501
I20260924 07:15:29.822476 21379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38501 every 8 connection(s)
I20260924 07:15:29.823328 21312 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:29.826174 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:29.832002 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36: Bootstrap starting.
I20260924 07:15:29.833621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21312
I20260924 07:15:29.833750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:29.834012 392 external_mini_cluster.cc:1428] Setting key 72796fd61824b0c0b8f699c8bad2764f
I20260924 07:15:29.834872 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:29.835819 21380 log.cc:824] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:29.837889 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36: No bootstrap required, opened a new log
I20260924 07:15:29.841519 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260924 07:15:29.841769 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:29.841840 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126dc3214d624fd2a4bc226a1fdd6d36, State: Initialized, Role: FOLLOWER
I20260924 07:15:29.842280 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260924 07:15:29.842427 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:15:29.842494 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:29.842593 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:29.843801 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260924 07:15:29.844215 21380 leader_election.cc:302] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 126dc3214d624fd2a4bc226a1fdd6d36; no voters:
I20260924 07:15:29.844594 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:29.844692 21385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:29.844906 21385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [term 1 LEADER]: Becoming Leader. State: Replica: 126dc3214d624fd2a4bc226a1fdd6d36, State: Running, Role: LEADER
I20260924 07:15:29.845325 21385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } }
I20260924 07:15:29.846011 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:29.846637 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 126dc3214d624fd2a4bc226a1fdd6d36. Latest consensus state: current_term: 1 leader_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } } }
I20260924 07:15:29.846761 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:29.847051 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "126dc3214d624fd2a4bc226a1fdd6d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38501 } } }
I20260924 07:15:29.847147 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:29.847802 21391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:29.849601 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:29.835115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:53891 (local address 127.0.0.1:38501)
0924 07:15:29.835646 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:29.835662 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:29.835698 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:29.836505 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:29.836531 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:29.836914 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:29.837240 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:29.838039 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:29.839038 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:29.839924 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:29.840427 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:29.840573 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:29.843095 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:29.843102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:29.843120 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:29.843154 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:29.845160 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:29.845822 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:29.845828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:29.845835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:29.845986 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:29.846376 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:29.846379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:29.846381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:29.847112 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:29.847469 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:29.848299 (+ 830us) server_negotiation.cc:299] Negotiation successful
0924 07:15:29.848546 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":150,"threads_started":1}
I20260924 07:15:29.852747 21391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:29.852794 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:29.859802 21391 catalog_manager.cc:1347] Generated new cluster ID: 5ca1beb29a734d6986b53468bb207bf9
I20260924 07:15:29.860160 21391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:29.862872 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:29.854706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:48119 (local address 127.0.0.1:38501)
0924 07:15:29.854813 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:29.854818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:29.855319 (+ 501us) server_negotiation.cc:407] Connection header received
0924 07:15:29.855456 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:29.855459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:29.855508 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:29.855572 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:29.856238 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:29.856761 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:29.857546 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:29.857798 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:29.857886 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:29.859585 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:29.859588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:29.859591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:29.859620 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:29.861098 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:29.861604 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:29.861608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:29.861610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:29.861690 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:29.862024 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:29.862028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:29.862029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:29.862202 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:29.862355 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:29.862606 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 07:15:29.862699 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:15:29.881102 21391 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:29.882161 21391 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:29.887689 21391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 126dc3214d624fd2a4bc226a1fdd6d36: Generated new TSK 0
I20260924 07:15:29.888474 21391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:29.900205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21312
W20260924 07:15:29.904954 21297 connection.cc:535] client connection to 127.0.0.1:38501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38501): Connection reset by peer (error 104)
2026-09-24T07:15:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21411](info): set up 2 sockets
Sep 24 07:15:29 dist-test-slave-cnv3 krb5kdc[21411](info): commencing operation
krb5kdc: starting...
W20260924 07:15:30.859413 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:30 dist-test-slave-cnv3 krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:30.982336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46731
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:31.093094 21424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:31.093385 21424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:31.093444 21424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:31.097551 21424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:31.097627 21424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:31.097653 21424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:31.097674 21424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:31.097703 21424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:31.103534 21424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:31.104707 21424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:31.105759 21424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:31.114686 21430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:31.114650 21429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:31.114745 21424 server_base.cc:1034] running on GCE node
W20260924 07:15:31.114669 21432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:31.115612 21424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:31.116811 21424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:31.117969 21424 hybrid_clock.cc:648] HybridClock initialized: now 1790234131117942 us; error 42 us; skew 500 ppm
Sep 24 07:15:31 dist-test-slave-cnv3 krb5kdc[21411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:31.125197 21424 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:31.126226 21424 webserver.cc:460] Webserver started at http://127.0.0.1:45445/ using document root <none> and password file <none>
I20260924 07:15:31.126778 21424 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:31.126854 21424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:31.127043 21424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:31.129155 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "804df39c1fe94b139e8e441bcfec599c"
format_stamp: "Formatted at 2026-09-24 07:15:31 on dist-test-slave-cnv3"
server_key: "d8e03dc91ee3fad8f0191ef0e9f3fcd7"
server_key_iv: "d8e344c26bfb9189e40e5fafb6b533be"
server_key_version: "encryptionkey@0"
I20260924 07:15:31.129689 21424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "804df39c1fe94b139e8e441bcfec599c"
format_stamp: "Formatted at 2026-09-24 07:15:31 on dist-test-slave-cnv3"
server_key: "d8e03dc91ee3fad8f0191ef0e9f3fcd7"
server_key_iv: "d8e344c26bfb9189e40e5fafb6b533be"
server_key_version: "encryptionkey@0"
I20260924 07:15:31.133476 21424 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 07:15:31.135970 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:31.137205 21424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:31.137360 21424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "804df39c1fe94b139e8e441bcfec599c"
format_stamp: "Formatted at 2026-09-24 07:15:31 on dist-test-slave-cnv3"
server_key: "d8e03dc91ee3fad8f0191ef0e9f3fcd7"
server_key_iv: "d8e344c26bfb9189e40e5fafb6b533be"
server_key_version: "encryptionkey@0"
I20260924 07:15:31.137470 21424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:31.163461 21424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:31.166692 21424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:31.166888 21424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:31.173720 21424 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46731
I20260924 07:15:31.173746 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46731 every 8 connection(s)
I20260924 07:15:31.174671 21424 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:31.177479 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:31.178819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21424
I20260924 07:15:31.178956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:31.179201 392 external_mini_cluster.cc:1428] Setting key f2ca17e334c9d0f2da3334dac3d9d6fd
I20260924 07:15:31.183552 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: Bootstrap starting.
I20260924 07:15:31.186009 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:31.186761 21492 log.cc:824] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:31.188376 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:31.180611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:57509 (local address 127.0.0.1:46731)
0924 07:15:31.181072 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:31.181087 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:31.181122 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:31.181768 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:31.181787 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:31.182126 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:31.182431 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:31.182830 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:31.183929 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:31.184785 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:31.185381 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:31.185573 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:31.186506 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:31.186511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:31.186523 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:31.186904 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:31.186967 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:31.186990 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:31.187097 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:31.187288 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:15:31.187448 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":138,"threads_started":1}
I20260924 07:15:31.188987 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: No bootstrap required, opened a new log
I20260924 07:15:31.191533 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:31.193367 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260924 07:15:31.193678 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:31.193742 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804df39c1fe94b139e8e441bcfec599c, State: Initialized, Role: FOLLOWER
I20260924 07:15:31.194236 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260924 07:15:31.194401 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:31.194473 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:31.194583 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:31.195669 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260924 07:15:31.196017 21492 leader_election.cc:302] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804df39c1fe94b139e8e441bcfec599c; no voters:
I20260924 07:15:31.196319 21492 leader_election.cc:288] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:31.196403 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:31.196595 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [term 1 LEADER]: Becoming Leader. State: Replica: 804df39c1fe94b139e8e441bcfec599c, State: Running, Role: LEADER
I20260924 07:15:31.196952 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } }
I20260924 07:15:31.197685 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:31.198210 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:31.193953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:51521 (local address 127.0.0.1:46731)
0924 07:15:31.194079 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:31.194083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:31.194271 (+ 188us) server_negotiation.cc:407] Connection header received
0924 07:15:31.194375 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:31.194378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:31.194435 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:31.194529 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:31.194854 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:31.195463 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:31.196326 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:31.196670 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:31.197383 (+ 713us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:31.197754 (+ 371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:31.197758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:31.197761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:31.197774 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:31.197795 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:31.197806 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:31.197884 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:31.198025 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 07:15:31.198060 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:15:31.198750 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804df39c1fe94b139e8e441bcfec599c. Latest consensus state: current_term: 1 leader_uuid: "804df39c1fe94b139e8e441bcfec599c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } } }
I20260924 07:15:31.199079 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:31.198982 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804df39c1fe94b139e8e441bcfec599c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804df39c1fe94b139e8e441bcfec599c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46731 } } }
I20260924 07:15:31.199344 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:31.202509 21517 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:31.202597 21517 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:31.202705 21513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:31.203469 21513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:31.209091 21513 catalog_manager.cc:1347] Generated new cluster ID: a09ef60c407c4399b0d27fcabf23454a
I20260924 07:15:31.209180 21513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:31.244040 21513 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:31.245380 21513 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:31.252213 21513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804df39c1fe94b139e8e441bcfec599c: Generated new TSK 0
I20260924 07:15:31.253136 21513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:31.258997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21424
W20260924 07:15:31.264236 21409 connection.cc:535] client connection to 127.0.0.1:46731 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46731): Connection reset by peer (error 104)
2026-09-24T07:15:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:31 dist-test-slave-cnv3 krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:31 dist-test-slave-cnv3 krb5kdc[21523](info): set up 2 sockets
Sep 24 07:15:31 dist-test-slave-cnv3 krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260924 07:15:32.227442 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:32 dist-test-slave-cnv3 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:32.346707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:32.456773 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:32.457070 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:32.457115 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:32.461180 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:32.461253 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:32.461272 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:32.461287 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:32.461301 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:32.466928 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:32.468060 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:32.469051 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:32.477550 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:32.477574 21536 server_base.cc:1034] running on GCE node
W20260924 07:15:32.477511 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:32.477515 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:32.478458 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:32.479697 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:32.480897 21536 hybrid_clock.cc:648] HybridClock initialized: now 1790234132480867 us; error 47 us; skew 500 ppm
Sep 24 07:15:32 dist-test-slave-cnv3 krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:32.483691 21536 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:32.484946 21536 webserver.cc:460] Webserver started at http://127.0.0.1:42857/ using document root <none> and password file <none>
I20260924 07:15:32.485595 21536 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:32.485672 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:32.485882 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:32.488118 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "290c9034dbb242ba84deb592ef9c1b57"
format_stamp: "Formatted at 2026-09-24 07:15:32 on dist-test-slave-cnv3"
server_key: "db499d2ca683cacf590c58ccda7b5f97"
server_key_iv: "1c11e453d60389b0bc2d9302ed245ba9"
server_key_version: "encryptionkey@0"
I20260924 07:15:32.488721 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "290c9034dbb242ba84deb592ef9c1b57"
format_stamp: "Formatted at 2026-09-24 07:15:32 on dist-test-slave-cnv3"
server_key: "db499d2ca683cacf590c58ccda7b5f97"
server_key_iv: "1c11e453d60389b0bc2d9302ed245ba9"
server_key_version: "encryptionkey@0"
I20260924 07:15:32.492663 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:15:32.495167 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:32.496399 21536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:15:32.496518 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "290c9034dbb242ba84deb592ef9c1b57"
format_stamp: "Formatted at 2026-09-24 07:15:32 on dist-test-slave-cnv3"
server_key: "db499d2ca683cacf590c58ccda7b5f97"
server_key_iv: "1c11e453d60389b0bc2d9302ed245ba9"
server_key_version: "encryptionkey@0"
I20260924 07:15:32.496620 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:32.501322 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:32.504434 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:32.504604 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:32.511282 21536 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33743
I20260924 07:15:32.511300 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33743 every 8 connection(s)
I20260924 07:15:32.512207 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:32.512707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21536
I20260924 07:15:32.512840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:32.513072 392 external_mini_cluster.cc:1428] Setting key f163b7068ca9e0e5732672e6f05175bd
I20260924 07:15:32.515285 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:32.518458 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:32.514468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:37805 (local address 127.0.0.1:33743)
0924 07:15:32.514874 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:32.514891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:32.514933 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:15:32.515775 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:32.515797 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:32.516152 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:32.516514 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:32.517132 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:32.517138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:32.517151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:32.517163 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:32.517220 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:32.517242 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:32.517376 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:32.517437 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 07:15:32.517584 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260924 07:15:32.521818 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:32.522136 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: Bootstrap starting.
I20260924 07:15:32.524463 21604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:32.525157 21604 log.cc:824] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:32.526189 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:32.523859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:35165 (local address 127.0.0.1:33743)
0924 07:15:32.524066 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:32.524069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:32.524810 (+ 741us) server_negotiation.cc:407] Connection header received
0924 07:15:32.524989 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:32.524993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:32.525033 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:32.525104 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:32.525812 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:32.525816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:32.525818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:32.525830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:32.525859 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:32.525873 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:32.525955 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:32.526039 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:15:32.526073 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:32.527354 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: No bootstrap required, opened a new log
I20260924 07:15:32.530987 21604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } }
I20260924 07:15:32.531231 21604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:32.531288 21604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290c9034dbb242ba84deb592ef9c1b57, State: Initialized, Role: FOLLOWER
I20260924 07:15:32.531776 21604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } }
I20260924 07:15:32.531929 21604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:32.531993 21604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:32.532096 21604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:32.533030 21604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } }
I20260924 07:15:32.533354 21604 leader_election.cc:302] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 290c9034dbb242ba84deb592ef9c1b57; no voters:
I20260924 07:15:32.533658 21604 leader_election.cc:288] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:32.533773 21614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:32.534065 21614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [term 1 LEADER]: Becoming Leader. State: Replica: 290c9034dbb242ba84deb592ef9c1b57, State: Running, Role: LEADER
I20260924 07:15:32.534365 21614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } }
I20260924 07:15:32.534816 21604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:32.536034 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 290c9034dbb242ba84deb592ef9c1b57. Latest consensus state: current_term: 1 leader_uuid: "290c9034dbb242ba84deb592ef9c1b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } } }
I20260924 07:15:32.536036 21615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "290c9034dbb242ba84deb592ef9c1b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290c9034dbb242ba84deb592ef9c1b57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33743 } } }
I20260924 07:15:32.536183 21615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:32.536183 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57 [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:32.539356 21629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:32.539458 21629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:32.539517 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:32.540213 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:32.545513 21626 catalog_manager.cc:1347] Generated new cluster ID: 34bd1b9c51ff43e3861883d6cb752c18
I20260924 07:15:32.545581 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:32.557844 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:32.558184 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:32.566475 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 290c9034dbb242ba84deb592ef9c1b57: Generated new TSK 0
I20260924 07:15:32.567453 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:32.575657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21536
2026-09-24T07:15:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:32 dist-test-slave-cnv3 krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:32 dist-test-slave-cnv3 krb5kdc[21635](info): set up 2 sockets
Sep 24 07:15:32 dist-test-slave-cnv3 krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260924 07:15:33.543545 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:33 dist-test-slave-cnv3 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:33.666592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:33.778268 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:33.778564 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:33.778606 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:33.782660 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:33.782734 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:33.782754 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:33.782770 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:33.782784 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:33.788363 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:33.789474 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:33.790462 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:33.798810 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:33.798774 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:33.798797 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:33.799150 21648 server_base.cc:1034] running on GCE node
I20260924 07:15:33.799733 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:33.800792 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:33.801950 21648 hybrid_clock.cc:648] HybridClock initialized: now 1790234133801935 us; error 38 us; skew 500 ppm
Sep 24 07:15:33 dist-test-slave-cnv3 krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:33.804625 21648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:33.805513 21648 webserver.cc:460] Webserver started at http://127.0.0.1:41683/ using document root <none> and password file <none>
I20260924 07:15:33.806064 21648 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:33.806131 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:33.806317 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:33.808372 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "f348bf0ab1984f8ebc5c620a5c41249e"
format_stamp: "Formatted at 2026-09-24 07:15:33 on dist-test-slave-cnv3"
server_key: "988d80ec1e4dbf0fea789509bd97481d"
server_key_iv: "bd62cb4db3c4a7874b494d4aed40b0e1"
server_key_version: "encryptionkey@0"
I20260924 07:15:33.808912 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "f348bf0ab1984f8ebc5c620a5c41249e"
format_stamp: "Formatted at 2026-09-24 07:15:33 on dist-test-slave-cnv3"
server_key: "988d80ec1e4dbf0fea789509bd97481d"
server_key_iv: "bd62cb4db3c4a7874b494d4aed40b0e1"
server_key_version: "encryptionkey@0"
I20260924 07:15:33.812551 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 07:15:33.815021 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:33.816320 21648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:15:33.816457 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "f348bf0ab1984f8ebc5c620a5c41249e"
format_stamp: "Formatted at 2026-09-24 07:15:33 on dist-test-slave-cnv3"
server_key: "988d80ec1e4dbf0fea789509bd97481d"
server_key_iv: "bd62cb4db3c4a7874b494d4aed40b0e1"
server_key_version: "encryptionkey@0"
I20260924 07:15:33.816557 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:33.833042 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:33.836161 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:33.836356 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:33.843673 21648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44495
I20260924 07:15:33.843701 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44495 every 8 connection(s)
I20260924 07:15:33.844643 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:33.847594 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:33.853199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21648
I20260924 07:15:33.853277 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e: Bootstrap starting.
I20260924 07:15:33.853405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:33.853636 392 external_mini_cluster.cc:1428] Setting key b2a7aac634679525c052bf2397bd6237
I20260924 07:15:33.855800 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:33.856518 21716 log.cc:824] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:33.858479 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e: No bootstrap required, opened a new log
Sep 24 07:15:33 dist-test-slave-cnv3 krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:15:33.862991 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260924 07:15:33.863317 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:33.863418 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f348bf0ab1984f8ebc5c620a5c41249e, State: Initialized, Role: FOLLOWER
I20260924 07:15:33.863986 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260924 07:15:33.864169 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:33.864243 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:33.864351 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:33.865337 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260924 07:15:33.865710 21716 leader_election.cc:302] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f348bf0ab1984f8ebc5c620a5c41249e; no voters:
I20260924 07:15:33.866044 21716 leader_election.cc:288] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:33.866140 21721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:33.866304 21721 raft_consensus.cc:686] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [term 1 LEADER]: Becoming Leader. State: Replica: f348bf0ab1984f8ebc5c620a5c41249e, State: Running, Role: LEADER
I20260924 07:15:33.866621 21721 consensus_queue.cc:237] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260924 07:15:33.867476 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:33.868290 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260924 07:15:33.868446 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:33.868793 21723 sys_catalog.cc:455] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f348bf0ab1984f8ebc5c620a5c41249e. Latest consensus state: current_term: 1 leader_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f348bf0ab1984f8ebc5c620a5c41249e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260924 07:15:33.868883 21723 sys_catalog.cc:458] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:33.869172 21730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:33.869572 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:33.855025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:33573 (local address 127.0.0.1:44495)
0924 07:15:33.855449 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:33.855465 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:33.855501 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:33.856158 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:33.856177 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:33.856538 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:33.856840 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:33.857630 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:33.858627 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:33.859771 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:33.860234 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:33.860401 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:33.863759 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:33.863767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:33.863786 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:33.863827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:33.865763 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:33.866435 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:33.866443 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:33.866450 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:33.866598 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:33.867041 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:33.867045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:33.867047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:33.867530 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:33.867735 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:33.868249 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:15:33.868490 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260924 07:15:33.872512 21730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:33.873524 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:33.879576 21730 catalog_manager.cc:1347] Generated new cluster ID: d6f690915e7e4b58835e8886deeeb6aa
I20260924 07:15:33.879662 21730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:33.883823 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:33.875869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:55609 (local address 127.0.0.1:44495)
0924 07:15:33.876125 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:33.876130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:33.876151 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:15:33.876339 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:33.876343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:33.876394 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:33.876441 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:33.877342 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:33.877995 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:33.878809 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:33.879103 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:33.879199 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:33.880632 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:33.880636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:33.880639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:33.880666 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:33.882021 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:33.882505 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:33.882508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:33.882510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:33.882598 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:33.882925 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:33.882928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:33.882930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:33.883114 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:33.883248 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:33.883563 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:15:33.883659 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260924 07:15:33.917528 21730 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:33.918522 21730 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:33.932392 21730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f348bf0ab1984f8ebc5c620a5c41249e: Generated new TSK 0
I20260924 07:15:33.933204 21730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:33.942837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21648
W20260924 07:15:33.947651 21633 connection.cc:535] client connection to 127.0.0.1:44495 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44495): Connection reset by peer (error 104)
2026-09-24T07:15:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:33 dist-test-slave-cnv3 krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:34 dist-test-slave-cnv3 krb5kdc[21747](info): set up 2 sockets
Sep 24 07:15:34 dist-test-slave-cnv3 krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260924 07:15:34.899519 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:34 dist-test-slave-cnv3 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:35.022797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:35.136215 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:35.136533 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:35.136595 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:35.140830 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:35.140918 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:35.140952 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:35.140974 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:35.140993 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:35.147010 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:35.148209 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:35.149281 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:35.158404 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:35.158373 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:35.158455 21760 server_base.cc:1034] running on GCE node
W20260924 07:15:35.158401 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:35.159260 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:35.160502 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:35.161690 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790234135161679 us; error 31 us; skew 500 ppm
Sep 24 07:15:35 dist-test-slave-cnv3 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:35.164705 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:35.165658 21760 webserver.cc:460] Webserver started at http://127.0.0.1:37393/ using document root <none> and password file <none>
I20260924 07:15:35.166246 21760 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:35.166318 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:35.166520 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:35.168782 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "582f4b631427477b91cdccb60f2479ab"
format_stamp: "Formatted at 2026-09-24 07:15:35 on dist-test-slave-cnv3"
server_key: "c17d974c09b7831958a9125a4ea49117"
server_key_iv: "25df7e432fb0f00a5baa14dc38e2c899"
server_key_version: "encryptionkey@0"
I20260924 07:15:35.169387 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "582f4b631427477b91cdccb60f2479ab"
format_stamp: "Formatted at 2026-09-24 07:15:35 on dist-test-slave-cnv3"
server_key: "c17d974c09b7831958a9125a4ea49117"
server_key_iv: "25df7e432fb0f00a5baa14dc38e2c899"
server_key_version: "encryptionkey@0"
I20260924 07:15:35.173599 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:35.176234 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:35.177570 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:35.177733 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "582f4b631427477b91cdccb60f2479ab"
format_stamp: "Formatted at 2026-09-24 07:15:35 on dist-test-slave-cnv3"
server_key: "c17d974c09b7831958a9125a4ea49117"
server_key_iv: "25df7e432fb0f00a5baa14dc38e2c899"
server_key_version: "encryptionkey@0"
I20260924 07:15:35.177860 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:35.203262 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:35.206324 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:35.206511 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:35.213560 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33127
I20260924 07:15:35.213627 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33127 every 8 connection(s)
I20260924 07:15:35.214663 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:35.217504 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:35.219666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21760
I20260924 07:15:35.219815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:35.220081 392 external_mini_cluster.cc:1428] Setting key eb57bd66239da93372833870648ebb3d
I20260924 07:15:35.223939 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: Bootstrap starting.
I20260924 07:15:35.226336 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:35.227110 21828 log.cc:824] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:35.229759 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: No bootstrap required, opened a new log
I20260924 07:15:35.230185 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:35.221524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:39463 (local address 127.0.0.1:33127)
0924 07:15:35.222024 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:35.222040 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:35.222078 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:15:35.222950 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:35.222981 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:35.223457 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:35.223835 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:35.224277 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:35.225284 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:35.226221 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:35.226750 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:35.226913 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:35.228036 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:35.228042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:35.228057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:35.228587 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:35.228658 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:35.228688 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:35.228902 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:35.229095 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:15:35.229235 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":154,"threads_started":1}
I20260924 07:15:35.233250 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:35.234032 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } }
I20260924 07:15:35.234295 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:35.234411 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582f4b631427477b91cdccb60f2479ab, State: Initialized, Role: FOLLOWER
I20260924 07:15:35.234859 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } }
I20260924 07:15:35.234982 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:35.235041 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:35.235139 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:35.236418 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } }
I20260924 07:15:35.236795 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 582f4b631427477b91cdccb60f2479ab; no voters:
I20260924 07:15:35.237123 21828 leader_election.cc:288] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:35.237237 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:35.237594 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [term 1 LEADER]: Becoming Leader. State: Replica: 582f4b631427477b91cdccb60f2479ab, State: Running, Role: LEADER
I20260924 07:15:35.237983 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } }
I20260924 07:15:35.238416 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:35.239637 21840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 582f4b631427477b91cdccb60f2479ab. Latest consensus state: current_term: 1 leader_uuid: "582f4b631427477b91cdccb60f2479ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } } }
I20260924 07:15:35.239773 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:35.235190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:38407 (local address 127.0.0.1:33127)
0924 07:15:35.235311 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:35.235315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:35.235529 (+ 214us) server_negotiation.cc:407] Connection header received
0924 07:15:35.235632 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:35.235635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:35.235685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:35.235774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:35.236018 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:35.236756 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:35.237804 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:35.238135 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:35.238209 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:35.239264 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:35.239268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:35.239271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:35.239286 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:35.239315 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:35.239327 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:35.239446 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:35.239545 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:15:35.239578 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:15:35.239962 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "582f4b631427477b91cdccb60f2479ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "582f4b631427477b91cdccb60f2479ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33127 } } }
I20260924 07:15:35.240032 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:35.239790 21840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:35.243275 21853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:35.243356 21853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:35.243450 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:35.244335 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:35.251196 21850 catalog_manager.cc:1347] Generated new cluster ID: 5f0f152b9b084e089d2ff9554a93d2a9
I20260924 07:15:35.251287 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:35.261441 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:35.263312 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:35.271706 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 582f4b631427477b91cdccb60f2479ab: Generated new TSK 0
I20260924 07:15:35.272616 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:35.282012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21760
W20260924 07:15:35.286597 21745 connection.cc:535] client connection to 127.0.0.1:33127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33127): Connection reset by peer (error 104)
2026-09-24T07:15:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:35 dist-test-slave-cnv3 krb5kdc[21859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:35 dist-test-slave-cnv3 krb5kdc[21859](info): set up 2 sockets
Sep 24 07:15:35 dist-test-slave-cnv3 krb5kdc[21859](info): commencing operation
krb5kdc: starting...
W20260924 07:15:36.247300 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:36 dist-test-slave-cnv3 krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:15:36.371910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:36.481375 21872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:36.481673 21872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:36.481715 21872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:36.485765 21872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:36.485829 21872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:36.485847 21872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:36.485862 21872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:36.485877 21872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:36.491264 21872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39029
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:36.492352 21872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:36.493423 21872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:36.501793 21872 server_base.cc:1034] running on GCE node
W20260924 07:15:36.501678 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:36.501725 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:36.501678 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:36.502533 21872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:36.503679 21872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:36.504832 21872 hybrid_clock.cc:648] HybridClock initialized: now 1790234136504811 us; error 35 us; skew 500 ppm
Sep 24 07:15:36 dist-test-slave-cnv3 krb5kdc[21859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:36.511160 21872 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:15:36.512135 21872 webserver.cc:460] Webserver started at http://127.0.0.1:34051/ using document root <none> and password file <none>
I20260924 07:15:36.512669 21872 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:36.512734 21872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:36.512926 21872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:36.514904 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "eae0018ceeb14fd3b16ef6d22c36d21f"
format_stamp: "Formatted at 2026-09-24 07:15:36 on dist-test-slave-cnv3"
server_key: "25c77e1f0731aeb00e52dd9b95b69849"
server_key_iv: "456bf9a2ee022144da707bde15793824"
server_key_version: "encryptionkey@0"
I20260924 07:15:36.515472 21872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "eae0018ceeb14fd3b16ef6d22c36d21f"
format_stamp: "Formatted at 2026-09-24 07:15:36 on dist-test-slave-cnv3"
server_key: "25c77e1f0731aeb00e52dd9b95b69849"
server_key_iv: "456bf9a2ee022144da707bde15793824"
server_key_version: "encryptionkey@0"
I20260924 07:15:36.519011 21872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:15:36.521361 21887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:36.522562 21872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:15:36.522671 21872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "eae0018ceeb14fd3b16ef6d22c36d21f"
format_stamp: "Formatted at 2026-09-24 07:15:36 on dist-test-slave-cnv3"
server_key: "25c77e1f0731aeb00e52dd9b95b69849"
server_key_iv: "456bf9a2ee022144da707bde15793824"
server_key_version: "encryptionkey@0"
I20260924 07:15:36.522769 21872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:36.527179 21872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:36.530434 21872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:36.530633 21872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:36.537407 21872 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39029
I20260924 07:15:36.537431 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39029 every 8 connection(s)
I20260924 07:15:36.538316 21872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:36.541101 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:36.546783 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: Bootstrap starting.
I20260924 07:15:36.547802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21872
I20260924 07:15:36.547951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:36.548240 392 external_mini_cluster.cc:1428] Setting key 0fed54352d1b849a2478f7b1bf9cb263
I20260924 07:15:36.549892 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:36.550616 21940 log.cc:824] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:36.552861 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: No bootstrap required, opened a new log
I20260924 07:15:36.553556 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:36.549426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:50357 (local address 127.0.0.1:39029)
0924 07:15:36.549899 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:36.549915 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:36.549954 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:15:36.550674 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:36.550697 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:36.551048 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:36.551478 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:36.552273 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:36.552277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:36.552289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:36.552299 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:36.552353 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:36.552371 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:36.552460 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:36.552582 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 07:15:36.552735 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":149,"threads_started":1}
I20260924 07:15:36.556305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:36.556538 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260924 07:15:36.556819 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:36.556890 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae0018ceeb14fd3b16ef6d22c36d21f, State: Initialized, Role: FOLLOWER
I20260924 07:15:36.557364 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260924 07:15:36.557526 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:36.557597 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:36.557703 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:36.558859 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260924 07:15:36.559247 21940 leader_election.cc:302] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae0018ceeb14fd3b16ef6d22c36d21f; no voters:
I20260924 07:15:36.559592 21940 leader_election.cc:288] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:36.559791 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:36.560184 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [term 1 LEADER]: Becoming Leader. State: Replica: eae0018ceeb14fd3b16ef6d22c36d21f, State: Running, Role: LEADER
I20260924 07:15:36.560415 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:36.558613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.152:56525 (local address 127.0.0.1:39029)
0924 07:15:36.558750 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:36.558755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:36.558912 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:15:36.559043 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:36.559045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:36.559090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:36.559153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:36.559975 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:36.559979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:36.559981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:36.559995 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:36.560026 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:36.560039 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:36.560126 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:36.560254 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:15:36.560303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:15:36.560653 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } }
I20260924 07:15:36.560981 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:36.562472 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260924 07:15:36.562515 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae0018ceeb14fd3b16ef6d22c36d21f. Latest consensus state: current_term: 1 leader_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae0018ceeb14fd3b16ef6d22c36d21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39029 } } }
I20260924 07:15:36.562651 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:36.562691 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:36.565534 21965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:36.565615 21965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:36.565696 21964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:36.566390 21964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:36.572356 21964 catalog_manager.cc:1347] Generated new cluster ID: 2c69a8b5ddb642d2a93bfed5ecd8c43f
I20260924 07:15:36.572451 21964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:36.603600 21964 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:36.603922 21964 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:36.612473 21964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eae0018ceeb14fd3b16ef6d22c36d21f: Generated new TSK 0
I20260924 07:15:36.613173 21964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:36.617301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21872
2026-09-24T07:15:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:36 dist-test-slave-cnv3 krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:36 dist-test-slave-cnv3 krb5kdc[21971](info): set up 2 sockets
Sep 24 07:15:36 dist-test-slave-cnv3 krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260924 07:15:37.595410 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:37 dist-test-slave-cnv3 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:37.718973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34105
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:37.830899 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:37.831240 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:37.831307 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:37.836011 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:37.836100 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:37.836127 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:37.836151 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:37.836179 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:37.842003 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:37.843180 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:37.844287 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:37.853076 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:37.853062 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:37.853225 21984 server_base.cc:1034] running on GCE node
W20260924 07:15:37.853482 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:37.854081 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:37.855288 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:37.856458 21984 hybrid_clock.cc:648] HybridClock initialized: now 1790234137856432 us; error 44 us; skew 500 ppm
Sep 24 07:15:37 dist-test-slave-cnv3 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:37.859201 21984 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:37.860169 21984 webserver.cc:460] Webserver started at http://127.0.98.62:45057/ using document root <none> and password file <none>
I20260924 07:15:37.860795 21984 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:37.860862 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:37.861034 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:37.863163 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "b9535608645445468042edf17c944197"
format_stamp: "Formatted at 2026-09-24 07:15:37 on dist-test-slave-cnv3"
server_key: "4db0ae041bca4d6e6d63cf604fbc8c07"
server_key_iv: "f8df2c4fd9bda98cc2dccb8de15d7ce3"
server_key_version: "encryptionkey@0"
I20260924 07:15:37.863796 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9535608645445468042edf17c944197"
format_stamp: "Formatted at 2026-09-24 07:15:37 on dist-test-slave-cnv3"
server_key: "4db0ae041bca4d6e6d63cf604fbc8c07"
server_key_iv: "f8df2c4fd9bda98cc2dccb8de15d7ce3"
server_key_version: "encryptionkey@0"
I20260924 07:15:37.868125 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:15:37.871171 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:37.872954 21984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:37.873099 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "b9535608645445468042edf17c944197"
format_stamp: "Formatted at 2026-09-24 07:15:37 on dist-test-slave-cnv3"
server_key: "4db0ae041bca4d6e6d63cf604fbc8c07"
server_key_iv: "f8df2c4fd9bda98cc2dccb8de15d7ce3"
server_key_version: "encryptionkey@0"
I20260924 07:15:37.873242 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:37.886948 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:37.890405 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:37.890605 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:37.897544 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34105
I20260924 07:15:37.897569 22051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34105 every 8 connection(s)
I20260924 07:15:37.898543 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:37.901561 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:37.905409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 21984
I20260924 07:15:37.905532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:37.905815 392 external_mini_cluster.cc:1428] Setting key 679a842e31e067444749e54a6596a62d
I20260924 07:15:37.908056 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197: Bootstrap starting.
I20260924 07:15:37.910318 22052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:37.911059 22052 log.cc:824] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:37.912988 22052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197: No bootstrap required, opened a new log
Sep 24 07:15:37 dist-test-slave-cnv3 krb5kdc[21971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:37.916903 22052 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 07:15:37.917138 22052 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:37.917172 22052 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9535608645445468042edf17c944197, State: Initialized, Role: FOLLOWER
I20260924 07:15:37.917619 22052 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 07:15:37.917744 22052 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:37.917811 22052 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:37.917901 22052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:37.918931 22052 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 07:15:37.919289 22052 leader_election.cc:302] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9535608645445468042edf17c944197; no voters:
I20260924 07:15:37.919605 22052 leader_election.cc:288] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:37.919721 22057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:37.919951 22057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [term 1 LEADER]: Becoming Leader. State: Replica: b9535608645445468042edf17c944197, State: Running, Role: LEADER
I20260924 07:15:37.920254 22057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } }
I20260924 07:15:37.920395 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:37.907414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48351 (local address 127.0.98.62:34105)
0924 07:15:37.907898 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:37.907913 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:37.907951 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:15:37.908699 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:37.908718 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:37.909116 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:37.909455 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:37.910242 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:37.911286 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:37.912219 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:37.912694 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:37.912837 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:37.915488 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:37.915494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:37.915510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:37.915538 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:37.917101 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:37.917658 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:37.917664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:37.917668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:37.917805 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:37.918165 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:37.918168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:37.918169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:37.918573 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:37.918738 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:37.919082 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 07:15:37.919327 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260924 07:15:37.920853 22052 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:37.921486 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9535608645445468042edf17c944197. Latest consensus state: current_term: 1 leader_uuid: "b9535608645445468042edf17c944197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } } }
I20260924 07:15:37.921599 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:37.921893 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9535608645445468042edf17c944197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9535608645445468042edf17c944197" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34105 } } }
I20260924 07:15:37.921983 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:37.922582 22064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:37.923623 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:37.926198 22064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:37.933210 22064 catalog_manager.cc:1347] Generated new cluster ID: cf7957058c7047a39999339e422d476f
I20260924 07:15:37.933293 22064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:37.934365 22055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:37.925680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40263 (local address 127.0.98.62:34105)
0924 07:15:37.925782 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:37.925787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:37.925993 (+ 206us) server_negotiation.cc:407] Connection header received
0924 07:15:37.926100 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:37.926103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:37.926150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:37.926229 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:37.927041 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:37.927868 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:37.928884 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:37.929123 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:37.929214 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:37.930609 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:37.930612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:37.930614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:37.930639 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:37.932291 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:37.932805 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:37.932808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:37.932809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:37.932898 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:37.933263 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:37.933267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:37.933269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:37.933524 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:37.933648 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:37.934097 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 07:15:37.934215 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:15:37.974149 22064 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:37.975113 22064 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:37.981251 22064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9535608645445468042edf17c944197: Generated new TSK 0
I20260924 07:15:37.982010 22064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:37.987874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 21984
W20260924 07:15:37.992740 21969 connection.cc:535] client connection to 127.0.98.62:34105 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34105): Connection reset by peer (error 104)
2026-09-24T07:15:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:38 dist-test-slave-cnv3 krb5kdc[22083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:38 dist-test-slave-cnv3 krb5kdc[22083](info): set up 2 sockets
Sep 24 07:15:38 dist-test-slave-cnv3 krb5kdc[22083](info): commencing operation
krb5kdc: starting...
W20260924 07:15:38.947309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:39 dist-test-slave-cnv3 krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:39.071985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37985
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:39.183272 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:39.183630 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:39.183707 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:39.187986 22096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:39.188055 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:39.188082 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:39.188107 22096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:39.188127 22096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:39.193608 22096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:39.194717 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:39.195753 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:39.204015 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:39.203995 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:39.204020 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:39.204454 22096 server_base.cc:1034] running on GCE node
I20260924 07:15:39.205057 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:39.206219 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:39.207456 22096 hybrid_clock.cc:648] HybridClock initialized: now 1790234139207438 us; error 31 us; skew 500 ppm
Sep 24 07:15:39 dist-test-slave-cnv3 krb5kdc[22083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234139, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:39.210346 22096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:39.211320 22096 webserver.cc:460] Webserver started at http://127.0.98.62:40393/ using document root <none> and password file <none>
I20260924 07:15:39.211891 22096 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:39.211958 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:39.212142 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:39.214164 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b"
format_stamp: "Formatted at 2026-09-24 07:15:39 on dist-test-slave-cnv3"
server_key: "f94aaaea325f893c8f88d70152bb3f0f"
server_key_iv: "6b4f15b5cdd3635c4bb1d406e2baeac5"
server_key_version: "encryptionkey@0"
I20260924 07:15:39.214700 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b"
format_stamp: "Formatted at 2026-09-24 07:15:39 on dist-test-slave-cnv3"
server_key: "f94aaaea325f893c8f88d70152bb3f0f"
server_key_iv: "6b4f15b5cdd3635c4bb1d406e2baeac5"
server_key_version: "encryptionkey@0"
I20260924 07:15:39.218534 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:39.221259 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:39.222569 22096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:15:39.222697 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b"
format_stamp: "Formatted at 2026-09-24 07:15:39 on dist-test-slave-cnv3"
server_key: "f94aaaea325f893c8f88d70152bb3f0f"
server_key_iv: "6b4f15b5cdd3635c4bb1d406e2baeac5"
server_key_version: "encryptionkey@0"
I20260924 07:15:39.222831 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:39.246493 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:39.250468 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:39.250671 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:39.259359 22096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37985
I20260924 07:15:39.259367 22163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37985 every 8 connection(s)
I20260924 07:15:39.260567 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:39.263352 22164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:39.268451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22096
I20260924 07:15:39.268584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:39.268880 392 external_mini_cluster.cc:1428] Setting key d36080c01875a316a5a2fd2b78911525
I20260924 07:15:39.269757 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: Bootstrap starting.
I20260924 07:15:39.272300 22164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:39.273159 22164 log.cc:824] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:39.275089 22164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: No bootstrap required, opened a new log
I20260924 07:15:39.279701 22164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } }
I20260924 07:15:39.279983 22164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:39.280040 22164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525be3bc1bc34684aa17b1f9f3e5ac3b, State: Initialized, Role: FOLLOWER
I20260924 07:15:39.280479 22164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } }
I20260924 07:15:39.280609 22164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:39.280671 22164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:39.280766 22164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:39.280902 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:39.270446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52985 (local address 127.0.98.62:37985)
0924 07:15:39.270891 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:39.270909 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:39.270950 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:15:39.271816 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:39.271838 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:39.272276 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:39.272644 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:39.273504 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:39.275031 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:39.276295 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:39.277011 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:39.277168 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:39.278538 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:39.278544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:39.278563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:39.279104 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:39.279175 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:39.279207 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:39.279332 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:39.279646 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:15:39.279825 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":111,"threads_started":1}
I20260924 07:15:39.281908 22164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } }
I20260924 07:15:39.282275 22164 leader_election.cc:302] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525be3bc1bc34684aa17b1f9f3e5ac3b; no voters:
I20260924 07:15:39.282599 22164 leader_election.cc:288] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:39.282769 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:39.283174 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [term 1 LEADER]: Becoming Leader. State: Replica: 525be3bc1bc34684aa17b1f9f3e5ac3b, State: Running, Role: LEADER
I20260924 07:15:39.283597 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } }
I20260924 07:15:39.283906 22164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:39.284160 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:39.285471 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525be3bc1bc34684aa17b1f9f3e5ac3b. Latest consensus state: current_term: 1 leader_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } } }
I20260924 07:15:39.285588 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:39.285964 22170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525be3bc1bc34684aa17b1f9f3e5ac3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37985 } } }
I20260924 07:15:39.286298 22170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:39.288906 22188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:39.288985 22188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:39.289121 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:39.289826 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:39.295594 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:39.288199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33285 (local address 127.0.98.62:37985)
0924 07:15:39.288322 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:39.288327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:39.288772 (+ 445us) server_negotiation.cc:407] Connection header received
0924 07:15:39.288887 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:39.288891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:39.288943 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:39.289024 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:39.291714 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:39.292481 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:39.293610 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:39.293958 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:39.294046 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:39.294980 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:39.294984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:39.294987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:39.295001 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:39.295029 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:39.295043 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:39.295124 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:39.295301 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:15:39.295362 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:15:39.295928 22187 catalog_manager.cc:1347] Generated new cluster ID: 2abb13ab67fc42acb2a33f5e4e996873
I20260924 07:15:39.296005 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:39.335160 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:39.336524 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:39.346318 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 525be3bc1bc34684aa17b1f9f3e5ac3b: Generated new TSK 0
I20260924 07:15:39.347231 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:39.360904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22096
W20260924 07:15:39.365895 22081 connection.cc:535] client connection to 127.0.98.62:37985 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:37985): Connection reset by peer (error 104)
2026-09-24T07:15:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:39 dist-test-slave-cnv3 krb5kdc[22195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:39 dist-test-slave-cnv3 krb5kdc[22195](info): set up 2 sockets
Sep 24 07:15:39 dist-test-slave-cnv3 krb5kdc[22195](info): commencing operation
krb5kdc: starting...
W20260924 07:15:40.323427 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:40 dist-test-slave-cnv3 krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:40.446533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:40.556928 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:40.557209 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:40.557252 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:40.561311 22208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:40.561393 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:40.561412 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:40.561429 22208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:40.561442 22208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:40.566962 22208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:40.568068 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:40.569068 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:40.577519 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:40.577508 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:40.577505 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:40.577550 22208 server_base.cc:1034] running on GCE node
I20260924 07:15:40.578306 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:40.579435 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:40.580624 22208 hybrid_clock.cc:648] HybridClock initialized: now 1790234140580601 us; error 40 us; skew 500 ppm
Sep 24 07:15:40 dist-test-slave-cnv3 krb5kdc[22195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:40.583565 22208 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:40.584559 22208 webserver.cc:460] Webserver started at http://127.0.98.62:46237/ using document root <none> and password file <none>
I20260924 07:15:40.585114 22208 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:40.585181 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:40.585369 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:40.587483 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "fe82861f06994b2eb9150fad7e6354f8"
format_stamp: "Formatted at 2026-09-24 07:15:40 on dist-test-slave-cnv3"
server_key: "ba9e3089b0d0b077619d8d486f15a952"
server_key_iv: "80f165d431e05480a9ff8f9ce7e867ec"
server_key_version: "encryptionkey@0"
I20260924 07:15:40.588028 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe82861f06994b2eb9150fad7e6354f8"
format_stamp: "Formatted at 2026-09-24 07:15:40 on dist-test-slave-cnv3"
server_key: "ba9e3089b0d0b077619d8d486f15a952"
server_key_iv: "80f165d431e05480a9ff8f9ce7e867ec"
server_key_version: "encryptionkey@0"
I20260924 07:15:40.591778 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:40.594345 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:40.595630 22208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:40.595752 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "fe82861f06994b2eb9150fad7e6354f8"
format_stamp: "Formatted at 2026-09-24 07:15:40 on dist-test-slave-cnv3"
server_key: "ba9e3089b0d0b077619d8d486f15a952"
server_key_iv: "80f165d431e05480a9ff8f9ce7e867ec"
server_key_version: "encryptionkey@0"
I20260924 07:15:40.595856 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:40.600509 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:40.603490 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:40.603657 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:40.610385 22208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33885
I20260924 07:15:40.610409 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33885 every 8 connection(s)
I20260924 07:15:40.611296 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:40.612591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22208
I20260924 07:15:40.612702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:40.612963 392 external_mini_cluster.cc:1428] Setting key 90b41aa39afa9a5d4bb7a762453f8378
I20260924 07:15:40.614231 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:40.618047 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:40.614375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49291 (local address 127.0.98.62:33885)
0924 07:15:40.614788 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:40.614802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:40.614837 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:40.615552 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:40.615570 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:40.615859 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:40.616168 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:40.616932 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:40.616936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:40.616947 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:40.616958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:40.617008 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:40.617026 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:40.617152 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:40.617220 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:15:40.617367 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":130,"threads_started":1}
I20260924 07:15:40.620188 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8: Bootstrap starting.
I20260924 07:15:40.620831 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:40.622449 22276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:40.623255 22276 log.cc:824] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:40.624439 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:40.622786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33241 (local address 127.0.98.62:33885)
0924 07:15:40.622891 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:40.622896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:40.623093 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:15:40.623200 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:40.623203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:40.623236 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:40.623300 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:40.624016 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:40.624021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:40.624024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:40.624036 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:40.624062 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:40.624089 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:40.624192 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:40.624271 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:15:40.624307 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:15:40.625350 22276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8: No bootstrap required, opened a new log
I20260924 07:15:40.629055 22276 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } }
I20260924 07:15:40.629285 22276 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:40.629319 22276 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe82861f06994b2eb9150fad7e6354f8, State: Initialized, Role: FOLLOWER
I20260924 07:15:40.629683 22276 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } }
I20260924 07:15:40.629789 22276 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:40.629845 22276 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:40.629956 22276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:40.630899 22276 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } }
I20260924 07:15:40.631206 22276 leader_election.cc:302] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe82861f06994b2eb9150fad7e6354f8; no voters:
I20260924 07:15:40.631520 22276 leader_election.cc:288] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:40.631697 22286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:40.631901 22286 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [term 1 LEADER]: Becoming Leader. State: Replica: fe82861f06994b2eb9150fad7e6354f8, State: Running, Role: LEADER
I20260924 07:15:40.632225 22286 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } }
I20260924 07:15:40.632735 22276 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:40.633816 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe82861f06994b2eb9150fad7e6354f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } } }
I20260924 07:15:40.633947 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:40.634264 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe82861f06994b2eb9150fad7e6354f8. Latest consensus state: current_term: 1 leader_uuid: "fe82861f06994b2eb9150fad7e6354f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe82861f06994b2eb9150fad7e6354f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33885 } } }
I20260924 07:15:40.634372 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:40.634318 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:40.637107 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:40.642565 22295 catalog_manager.cc:1347] Generated new cluster ID: a6951cb8888040cb8a6df77b5f38d4f5
I20260924 07:15:40.642688 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:40.657323 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:40.657706 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:40.665791 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe82861f06994b2eb9150fad7e6354f8: Generated new TSK 0
I20260924 07:15:40.666623 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:40.672834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22208
2026-09-24T07:15:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:40 dist-test-slave-cnv3 krb5kdc[22307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:40 dist-test-slave-cnv3 krb5kdc[22307](info): set up 2 sockets
Sep 24 07:15:40 dist-test-slave-cnv3 krb5kdc[22307](info): commencing operation
krb5kdc: starting...
W20260924 07:15:41.631475 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:41 dist-test-slave-cnv3 krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:41.762239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41905
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:41.873309 22320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:41.873624 22320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:41.873687 22320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:41.877861 22320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:41.877940 22320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:41.877967 22320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:41.877988 22320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:41.878019 22320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:41.883594 22320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:41.884721 22320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:41.885758 22320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:41.894577 22325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:41.894486 22326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:41.894483 22328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:41.894559 22320 server_base.cc:1034] running on GCE node
I20260924 07:15:41.895339 22320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:41.896522 22320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:41.897689 22320 hybrid_clock.cc:648] HybridClock initialized: now 1790234141897680 us; error 42 us; skew 500 ppm
Sep 24 07:15:41 dist-test-slave-cnv3 krb5kdc[22307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:41.906260 22320 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:41.907295 22320 webserver.cc:460] Webserver started at http://127.0.98.62:37313/ using document root <none> and password file <none>
I20260924 07:15:41.907883 22320 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:41.907954 22320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:41.908152 22320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:41.910226 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "c617520c19f743bf989d4d523540547a"
format_stamp: "Formatted at 2026-09-24 07:15:41 on dist-test-slave-cnv3"
server_key: "a866a4de303615cf0474e72b62bbf3cd"
server_key_iv: "e89b7ce85694f71479905f4529e4b815"
server_key_version: "encryptionkey@0"
I20260924 07:15:41.910797 22320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "c617520c19f743bf989d4d523540547a"
format_stamp: "Formatted at 2026-09-24 07:15:41 on dist-test-slave-cnv3"
server_key: "a866a4de303615cf0474e72b62bbf3cd"
server_key_iv: "e89b7ce85694f71479905f4529e4b815"
server_key_version: "encryptionkey@0"
I20260924 07:15:41.914631 22320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:41.917223 22335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:41.918470 22320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:15:41.918594 22320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "c617520c19f743bf989d4d523540547a"
format_stamp: "Formatted at 2026-09-24 07:15:41 on dist-test-slave-cnv3"
server_key: "a866a4de303615cf0474e72b62bbf3cd"
server_key_iv: "e89b7ce85694f71479905f4529e4b815"
server_key_version: "encryptionkey@0"
I20260924 07:15:41.918708 22320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:41.947258 22320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:41.950600 22320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:41.950793 22320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:41.957608 22320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41905
I20260924 07:15:41.957636 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41905 every 8 connection(s)
I20260924 07:15:41.958575 22320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:41.959012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22320
I20260924 07:15:41.959142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:41.959429 392 external_mini_cluster.cc:1428] Setting key 824c8ef41a1c3fe52e5ecd014891d9e7
I20260924 07:15:41.961772 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 07:15:41 dist-test-slave-cnv3 krb5kdc[22307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:41.968384 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a: Bootstrap starting.
I20260924 07:15:41.971149 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:41.972074 22388 log.cc:824] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:41.974339 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:41.960812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57021 (local address 127.0.98.62:41905)
0924 07:15:41.961221 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:41.961236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:41.961271 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:41.961913 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:41.961935 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:41.962278 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:41.962560 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:41.963469 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:41.964454 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:41.965622 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:41.966082 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:41.966193 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:41.969045 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:41.969052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:41.969069 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:41.969108 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:41.971068 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:41.971614 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:41.971621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:41.971626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:41.971781 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:41.972065 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:41.972069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:41.972070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:41.972615 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:41.972775 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:41.973088 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 07:15:41.973399 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260924 07:15:41.974501 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a: No bootstrap required, opened a new log
I20260924 07:15:41.977171 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:41.978538 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 07:15:41.978783 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:41.978842 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c617520c19f743bf989d4d523540547a, State: Initialized, Role: FOLLOWER
I20260924 07:15:41.979355 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 07:15:41.979553 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:41.979620 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:41.979719 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:41.980666 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 07:15:41.980996 22388 leader_election.cc:302] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c617520c19f743bf989d4d523540547a; no voters:
I20260924 07:15:41.981297 22388 leader_election.cc:288] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:41.981415 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:41.981585 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [term 1 LEADER]: Becoming Leader. State: Replica: c617520c19f743bf989d4d523540547a, State: Running, Role: LEADER
I20260924 07:15:41.981899 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } }
I20260924 07:15:41.982630 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:41.983105 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c617520c19f743bf989d4d523540547a. Latest consensus state: current_term: 1 leader_uuid: "c617520c19f743bf989d4d523540547a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } } }
I20260924 07:15:41.983217 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:41.983536 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c617520c19f743bf989d4d523540547a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617520c19f743bf989d4d523540547a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41905 } } }
I20260924 07:15:41.983641 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:41.984076 22403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:41.987509 22403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:41.987668 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:41.979068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40855 (local address 127.0.98.62:41905)
0924 07:15:41.979187 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:41.979190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:41.979346 (+ 156us) server_negotiation.cc:407] Connection header received
0924 07:15:41.979499 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:41.979501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:41.979550 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:41.979631 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:41.980347 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:41.980858 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:41.981841 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:41.982118 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:41.982207 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:41.983882 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:41.983886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:41.983889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:41.983915 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:41.985517 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:41.986021 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:41.986025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:41.986027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:41.986118 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:41.986411 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:41.986414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:41.986416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:41.986641 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:41.986770 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:41.987333 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 07:15:41.987479 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:15:41.993083 22403 catalog_manager.cc:1347] Generated new cluster ID: a52ae2073b7443ed9204d4f05c7756cb
I20260924 07:15:41.993161 22403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:42.003763 22403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:42.004706 22403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:42.010608 22403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c617520c19f743bf989d4d523540547a: Generated new TSK 0
I20260924 07:15:42.011289 22403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:42.018909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22320
2026-09-24T07:15:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:42 dist-test-slave-cnv3 krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:42 dist-test-slave-cnv3 krb5kdc[22419](info): set up 2 sockets
Sep 24 07:15:42 dist-test-slave-cnv3 krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260924 07:15:42.975304 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:43 dist-test-slave-cnv3 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:43.097828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:43.210143 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:43.210448 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:43.210505 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:43.214967 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:43.215061 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:43.215090 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:43.215119 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:43.215142 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:43.220803 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:43.221939 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:43.222983 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:43.231554 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:43.231586 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:43.231577 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:43.231976 22432 server_base.cc:1034] running on GCE node
I20260924 07:15:43.232625 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:43.233836 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:43.235031 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790234143235013 us; error 32 us; skew 500 ppm
Sep 24 07:15:43 dist-test-slave-cnv3 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:43.238050 22432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:43.239018 22432 webserver.cc:460] Webserver started at http://127.0.98.62:37171/ using document root <none> and password file <none>
I20260924 07:15:43.239650 22432 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:43.239718 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:43.239904 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:43.242074 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "231b08a335dd4ad9ac4e460367b17551"
format_stamp: "Formatted at 2026-09-24 07:15:43 on dist-test-slave-cnv3"
server_key: "05fe7394bf98052bcf3e3ddceca7f416"
server_key_iv: "b10c440551cbfc5db81c5a0f58cc4e5f"
server_key_version: "encryptionkey@0"
I20260924 07:15:43.242630 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "231b08a335dd4ad9ac4e460367b17551"
format_stamp: "Formatted at 2026-09-24 07:15:43 on dist-test-slave-cnv3"
server_key: "05fe7394bf98052bcf3e3ddceca7f416"
server_key_iv: "b10c440551cbfc5db81c5a0f58cc4e5f"
server_key_version: "encryptionkey@0"
I20260924 07:15:43.246477 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:15:43.249176 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:43.250419 22432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:43.250532 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "231b08a335dd4ad9ac4e460367b17551"
format_stamp: "Formatted at 2026-09-24 07:15:43 on dist-test-slave-cnv3"
server_key: "05fe7394bf98052bcf3e3ddceca7f416"
server_key_iv: "b10c440551cbfc5db81c5a0f58cc4e5f"
server_key_version: "encryptionkey@0"
I20260924 07:15:43.250658 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:43.261264 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:43.265167 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:43.265331 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:43.272226 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35877
I20260924 07:15:43.272269 22499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35877 every 8 connection(s)
I20260924 07:15:43.273211 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:43.273739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22432
I20260924 07:15:43.273873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:43.274123 392 external_mini_cluster.cc:1428] Setting key 2fd459be95b22f01e51417f6c68dde3c
I20260924 07:15:43.276367 22500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:43.281905 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: Bootstrap starting.
I20260924 07:15:43.282992 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:43.275535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57663 (local address 127.0.98.62:35877)
0924 07:15:43.276023 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:43.276038 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:43.276077 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:15:43.276797 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:43.276816 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:43.277170 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:43.277490 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:43.277977 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:43.278981 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:43.279933 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:43.280401 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:43.280547 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:43.281351 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:43.281356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:43.281368 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:43.281765 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:43.281821 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:43.281844 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:43.281927 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:43.282065 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 07:15:43.282275 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":157,"threads_started":1}
I20260924 07:15:43.284421 22500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:43.285234 22500 log.cc:824] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:43.285959 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:43.287760 22500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: No bootstrap required, opened a new log
I20260924 07:15:43.292357 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:43.288327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37083 (local address 127.0.98.62:35877)
0924 07:15:43.288442 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:43.288447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:43.288565 (+ 118us) server_negotiation.cc:407] Connection header received
0924 07:15:43.288661 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:43.288664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:43.288710 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:43.288801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:43.289025 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:43.289786 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:43.290571 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:43.290912 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:43.291020 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:43.291793 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:43.291797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:43.291799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:43.291814 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:43.291842 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:43.291855 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:43.291931 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:43.292070 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:15:43.292139 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:43.292192 22500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260924 07:15:43.292524 22500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:43.292578 22500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 231b08a335dd4ad9ac4e460367b17551, State: Initialized, Role: FOLLOWER
I20260924 07:15:43.293045 22500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260924 07:15:43.293187 22500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:43.293256 22500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:43.293361 22500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:43.294342 22500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260924 07:15:43.294667 22500 leader_election.cc:302] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 231b08a335dd4ad9ac4e460367b17551; no voters:
I20260924 07:15:43.294946 22500 leader_election.cc:288] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:43.295053 22510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:43.295272 22510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [term 1 LEADER]: Becoming Leader. State: Replica: 231b08a335dd4ad9ac4e460367b17551, State: Running, Role: LEADER
I20260924 07:15:43.295588 22510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } }
I20260924 07:15:43.296192 22500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:43.297226 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "231b08a335dd4ad9ac4e460367b17551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } } }
I20260924 07:15:43.297356 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:43.297626 22512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 231b08a335dd4ad9ac4e460367b17551. Latest consensus state: current_term: 1 leader_uuid: "231b08a335dd4ad9ac4e460367b17551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "231b08a335dd4ad9ac4e460367b17551" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35877 } } }
I20260924 07:15:43.297717 22512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551 [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:43.301138 22525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:43.301234 22525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:43.301331 22519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:43.302039 22519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:43.308027 22519 catalog_manager.cc:1347] Generated new cluster ID: 3ee69cdb9e234cd9a759e6e953246298
I20260924 07:15:43.308154 22519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:43.321996 22519 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:43.323199 22519 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:43.333323 22519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 231b08a335dd4ad9ac4e460367b17551: Generated new TSK 0
I20260924 07:15:43.334185 22519 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:43.349188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22432
2026-09-24T07:15:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:43 dist-test-slave-cnv3 krb5kdc[22531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:43 dist-test-slave-cnv3 krb5kdc[22531](info): set up 2 sockets
Sep 24 07:15:43 dist-test-slave-cnv3 krb5kdc[22531](info): commencing operation
krb5kdc: starting...
W20260924 07:15:44.311510 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:44 dist-test-slave-cnv3 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:44.434592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:44.544862 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:44.545174 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:44.545235 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:44.549461 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:44.549544 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:44.549571 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:44.549593 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:44.549612 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:44.555171 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:44.556348 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:44.557395 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:44.566282 22552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:44.566282 22549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:44.566282 22550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:44.566787 22544 server_base.cc:1034] running on GCE node
I20260924 07:15:44.567485 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:44.568702 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:44.569917 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790234144569895 us; error 35 us; skew 500 ppm
Sep 24 07:15:44 dist-test-slave-cnv3 krb5kdc[22531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:44.572844 22544 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:44.573858 22544 webserver.cc:460] Webserver started at http://127.0.98.62:40651/ using document root <none> and password file <none>
I20260924 07:15:44.574430 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:44.574503 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:44.574712 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:44.576869 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "aef9360777c14b4092c8354ec8b4c1bc"
format_stamp: "Formatted at 2026-09-24 07:15:44 on dist-test-slave-cnv3"
server_key: "a58de23e43bf8d7fc1be0bcaa0b015aa"
server_key_iv: "044e68124f0d4c1a3d931983f647de0c"
server_key_version: "encryptionkey@0"
I20260924 07:15:44.577437 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "aef9360777c14b4092c8354ec8b4c1bc"
format_stamp: "Formatted at 2026-09-24 07:15:44 on dist-test-slave-cnv3"
server_key: "a58de23e43bf8d7fc1be0bcaa0b015aa"
server_key_iv: "044e68124f0d4c1a3d931983f647de0c"
server_key_version: "encryptionkey@0"
I20260924 07:15:44.581384 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:15:44.584069 22559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:44.585501 22544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:44.585639 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "aef9360777c14b4092c8354ec8b4c1bc"
format_stamp: "Formatted at 2026-09-24 07:15:44 on dist-test-slave-cnv3"
server_key: "a58de23e43bf8d7fc1be0bcaa0b015aa"
server_key_iv: "044e68124f0d4c1a3d931983f647de0c"
server_key_version: "encryptionkey@0"
I20260924 07:15:44.585762 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:44.590517 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:44.593673 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:44.593871 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:44.600690 22544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35795
I20260924 07:15:44.600703 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35795 every 8 connection(s)
I20260924 07:15:44.601667 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:44.604455 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:44.609949 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc: Bootstrap starting.
I20260924 07:15:44.611253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22544
I20260924 07:15:44.611409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:44.611689 392 external_mini_cluster.cc:1428] Setting key 8fa7c8146995a755eb9421e08a9a3f80
I20260924 07:15:44.612774 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:44.613575 22612 log.cc:824] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:44.615681 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc: No bootstrap required, opened a new log
I20260924 07:15:44.616684 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:44.612802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40549 (local address 127.0.98.62:35795)
0924 07:15:44.613261 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:44.613275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:44.613310 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:44.614067 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:44.614095 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:44.614444 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:44.614752 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:44.615536 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:44.615541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:44.615552 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:44.615562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:44.615614 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:44.615634 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:44.615725 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:44.615860 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 07:15:44.616009 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":150,"threads_started":1}
I20260924 07:15:44.619285 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:44.619287 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260924 07:15:44.619553 22612 raft_consensus.cc:374] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:15:44.619608 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef9360777c14b4092c8354ec8b4c1bc, State: Initialized, Role: FOLLOWER
I20260924 07:15:44.620014 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260924 07:15:44.620144 22612 raft_consensus.cc:388] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:15:44.620206 22612 raft_consensus.cc:482] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:15:44.620296 22612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:44.621261 22612 raft_consensus.cc:504] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260924 07:15:44.621627 22612 leader_election.cc:302] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aef9360777c14b4092c8354ec8b4c1bc; no voters:
I20260924 07:15:44.621933 22612 leader_election.cc:288] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:15:44.622033 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:44.622202 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [term 1 LEADER]: Becoming Leader. State: Replica: aef9360777c14b4092c8354ec8b4c1bc, State: Running, Role: LEADER
I20260924 07:15:44.622507 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } }
I20260924 07:15:44.622861 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:44.621261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50593 (local address 127.0.98.62:35795)
0924 07:15:44.621382 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:44.621387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:44.621495 (+ 108us) server_negotiation.cc:407] Connection header received
0924 07:15:44.621605 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:44.621608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:44.621644 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:44.621721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:44.622494 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:44.622498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:44.622500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:15:44.622512 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:44.622542 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:15:44.622555 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:44.622637 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:44.622716 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:15:44.622752 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:15:44.623301 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:44.624279 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aef9360777c14b4092c8354ec8b4c1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } } }
I20260924 07:15:44.624410 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:44.624645 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader aef9360777c14b4092c8354ec8b4c1bc. Latest consensus state: current_term: 1 leader_uuid: "aef9360777c14b4092c8354ec8b4c1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aef9360777c14b4092c8354ec8b4c1bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35795 } } }
I20260924 07:15:44.624774 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:44.624776 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:44.627753 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:44.632987 22631 catalog_manager.cc:1347] Generated new cluster ID: b8f655237fbd49788437f659e8d932eb
I20260924 07:15:44.633078 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:44.645828 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:44.646245 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:44.655483 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aef9360777c14b4092c8354ec8b4c1bc: Generated new TSK 0
I20260924 07:15:44.656374 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:15:44.666360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22544
2026-09-24T07:15:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1317 ms)
[----------] 18 tests from AuthTokenIssuingTest (24145 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:44 dist-test-slave-cnv3 krb5kdc[22643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:44 dist-test-slave-cnv3 krb5kdc[22643](info): set up 2 sockets
Sep 24 07:15:44 dist-test-slave-cnv3 krb5kdc[22643](info): commencing operation
krb5kdc: starting...
W20260924 07:15:45.623420 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:45 dist-test-slave-cnv3 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:45.747850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:45.860453 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:45.860774 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:45.860798 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:45.860841 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:45.865259 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:45.865341 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:45.865361 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:45.865377 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:45.865392 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:45.870965 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:45.872098 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:45.873093 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:45.882277 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:45.882288 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:45.882344 22656 server_base.cc:1034] running on GCE node
W20260924 07:15:45.882288 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:45.883117 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:45.884330 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:45.885493 22656 hybrid_clock.cc:648] HybridClock initialized: now 1790234145885468 us; error 37 us; skew 500 ppm
Sep 24 07:15:45 dist-test-slave-cnv3 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:45.888275 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:45.889127 22656 webserver.cc:460] Webserver started at http://127.0.98.62:39935/ using document root <none> and password file <none>
I20260924 07:15:45.889652 22656 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:45.889711 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:45.889878 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:45.891963 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "1bf226c5f1e245daade783f8a0546266"
format_stamp: "Formatted at 2026-09-24 07:15:45 on dist-test-slave-cnv3"
server_key: "a4f53b5132137d080a055527d88e36f5"
server_key_iv: "e35838a8729e99441b28a4a15819f83e"
server_key_version: "encryptionkey@0"
I20260924 07:15:45.892468 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bf226c5f1e245daade783f8a0546266"
format_stamp: "Formatted at 2026-09-24 07:15:45 on dist-test-slave-cnv3"
server_key: "a4f53b5132137d080a055527d88e36f5"
server_key_iv: "e35838a8729e99441b28a4a15819f83e"
server_key_version: "encryptionkey@0"
I20260924 07:15:45.896219 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:45.898707 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:45.900077 22656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:45.900187 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "1bf226c5f1e245daade783f8a0546266"
format_stamp: "Formatted at 2026-09-24 07:15:45 on dist-test-slave-cnv3"
server_key: "a4f53b5132137d080a055527d88e36f5"
server_key_iv: "e35838a8729e99441b28a4a15819f83e"
server_key_version: "encryptionkey@0"
I20260924 07:15:45.900307 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:45.916975 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:45.920209 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:45.920370 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:45.926966 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44167
I20260924 07:15:45.927229 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44167 every 8 connection(s)
I20260924 07:15:45.928000 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:45.930888 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:45.934311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22656
I20260924 07:15:45.934418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:45.934648 392 external_mini_cluster.cc:1428] Setting key 8edf117b18395722202f7f0df2a41cdf
I20260924 07:15:45.934659 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 07:15:45 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:45.951653 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:45.937939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.98.62:44167)
0924 07:15:45.938230 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:45.938246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:45.938291 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:15:45.939019 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:45.939024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:45.939217 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:45.939409 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:45.940273 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:45.941149 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:45.942143 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:45.942538 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:45.942704 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:45.946833 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:45.946839 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:45.946852 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:45.946877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:45.948478 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:45.948935 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:45.948941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:45.948942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:45.949051 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:45.949356 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:45.949359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:45.949361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:45.949762 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:45.949879 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:45.950228 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 07:15:45.950525 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":191,"thread_start_us":72,"threads_started":1}
I20260924 07:15:45.951653 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:45.937173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.0.1:39874)
0924 07:15:45.937706 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:45.937752 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:45.938714 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:45.939421 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:45.939426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:45.939880 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:45.940122 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:45.940135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:45.941269 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:45.941272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:45.942025 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:45.942034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:45.942691 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:45.942696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:45.942797 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:45.943401 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:45.943425 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:45.946666 (+ 3241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:45.948587 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:45.948596 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:45.948607 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:45.948834 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:45.949135 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:45.949138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:45.949141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:45.949250 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:45.949871 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:45.949877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:45.950007 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 07:15:45.950155 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 07:15:45.950580 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":144,"thread_start_us":135,"threads_started":1}
I20260924 07:15:45.954063 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:45.954849 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:45.954526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33260)
0924 07:15:45.954656 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:45.954750 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:15:45.955070 22673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
I20260924 07:15:45.955964 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:45.955761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33276)
0924 07:15:45.955863 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:45.955907 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:15:45.956116 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:15:45.962414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42079
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:15:46.005355 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } attempt: 1
I20260924 07:15:46.006227 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.005912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33292)
0924 07:15:46.006064 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.006145 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 07:15:46.006944 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.006724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33298)
0924 07:15:46.006820 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.006886 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:15:46.007055 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
W20260924 07:15:46.073189 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:46.073513 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:46.073556 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:46.073621 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:15:46.075222 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } attempt: 2
I20260924 07:15:46.076097 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.075807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33314)
0924 07:15:46.075946 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.076006 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:15:46.076895 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.076684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33318)
0924 07:15:46.076778 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.076837 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:15:46.077023 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42079: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42079: connect: Connection refused (error 111)
W20260924 07:15:46.078091 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:46.078145 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:46.078174 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:46.078204 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:46.078233 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:46.083766 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:46.084864 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:46.085875 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:46.094290 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:46.094306 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:46.094405 22731 server_base.cc:1034] running on GCE node
W20260924 07:15:46.094589 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:46.095105 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:46.096253 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:46.097427 22731 hybrid_clock.cc:648] HybridClock initialized: now 1790234146097404 us; error 44 us; skew 500 ppm
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:46.100186 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:15:46.101159 22731 webserver.cc:460] Webserver started at http://127.0.98.61:34235/ using document root <none> and password file <none>
I20260924 07:15:46.101701 22731 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:46.101765 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:46.101967 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:46.104087 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data/instance:
uuid: "be488fcd23084d089ff53267555d47f4"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "b54d094aad508079c4e5a1fef1906b4a"
server_key_iv: "da1acf3acfd0c43304534ba1864dc375"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.104663 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal/instance:
uuid: "be488fcd23084d089ff53267555d47f4"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "b54d094aad508079c4e5a1fef1906b4a"
server_key_iv: "da1acf3acfd0c43304534ba1864dc375"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.108477 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:15:46.110970 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:46.112252 22731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:15:46.112392 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "be488fcd23084d089ff53267555d47f4"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "b54d094aad508079c4e5a1fef1906b4a"
server_key_iv: "da1acf3acfd0c43304534ba1864dc375"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.112499 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:46.126046 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:46.129143 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:46.129396 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:46.136117 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42079
I20260924 07:15:46.136468 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42079 every 8 connection(s)
I20260924 07:15:46.137087 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:15:46.138420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22731
I20260924 07:15:46.138543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-1/wal/instance
I20260924 07:15:46.138787 392 external_mini_cluster.cc:1428] Setting key 9f672360877aaa53eecf8bd4dbba4160
I20260924 07:15:46.140029 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:46.144014 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:46.159073 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.146029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.0.98.62:44167)
0924 07:15:46.146173 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.146177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.147109 (+ 932us) server_negotiation.cc:407] Connection header received
0924 07:15:46.148092 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.148094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.148144 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.148219 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.149655 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.150209 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.151798 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.152041 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.152123 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.155145 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.155149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.155151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.155180 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.156870 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.157444 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.157447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.157448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.157540 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.157963 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.157966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.157967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.158171 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.158267 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.158783 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.158907 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:15:46.159977 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.146397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.0.1:39886)
0924 07:15:46.146947 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.146995 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.147893 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.148385 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.148392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.148766 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.149490 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.149508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.150352 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.150356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.151677 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.151688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.152197 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.152201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.152417 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.153027 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.153049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.154929 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.157003 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.157012 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.157032 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.157324 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.157672 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.157675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.157676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.157864 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.158279 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.158286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.158563 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.158724 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.159028 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":158,"threads_started":1}
I20260924 07:15:46.161365 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:15:46.166436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43893
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:46.168206 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } attempt: 3
I20260924 07:15:46.170957 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.161831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33326)
0924 07:15:46.161973 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.161988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.162074 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.162664 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.162666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.162819 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.162911 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.162916 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.163942 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.163945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.164539 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.164549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.165139 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.165150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.165236 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.165689 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.165702 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.167049 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.169331 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.169334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.169335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.169512 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.169812 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.169815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.169817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.169902 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.170439 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.170443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.170549 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.170658 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.170787 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:15:46.171103 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.161992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.98.61:42079)
0924 07:15:46.162263 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.162272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.162328 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:15:46.162364 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.162368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.162520 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.162623 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.163001 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.163832 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.164634 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.165014 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.165095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.167192 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.167204 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.167223 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.167248 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.168969 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.169622 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.169629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.169630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.169728 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.169993 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.169996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.169998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.170339 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.170474 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.170755 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.170902 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260924 07:15:46.173254 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:46.173957 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.173719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35458)
0924 07:15:46.173826 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.173893 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:15:46.174139 22749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:46.174968 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.174721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35468)
0924 07:15:46.174873 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.174916 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 07:15:46.175104 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
I20260924 07:15:46.178475 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.168730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33328)
0924 07:15:46.169014 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.169030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.169111 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.169666 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.169670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.169847 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.170408 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.170421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.171447 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.171452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.172264 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.172288 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.172921 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.172925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.173042 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.173526 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.173541 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.175066 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.176936 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.176939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.176947 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.177236 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.177510 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.177512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.177514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.177581 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.178015 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.178018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.178128 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.178231 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.178355 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260924 07:15:46.178578 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.169094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.98.61:42079)
0924 07:15:46.169419 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.169427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.169439 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:15:46.169469 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.169472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.169510 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.169628 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.170537 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.171292 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.172421 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.172785 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.173287 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.175241 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.175244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.175247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.175285 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.176816 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.177334 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.177338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.177340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.177427 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.177656 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.177659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.177661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.177895 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.177992 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.178296 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.178414 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":126,"threads_started":1}
I20260924 07:15:46.179419 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:46.180075 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.179847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35474)
0924 07:15:46.179959 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.180013 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:15:46.180724 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.180496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35478)
0924 07:15:46.180630 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.180674 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 07:15:46.180860 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
I20260924 07:15:46.212015 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 1
I20260924 07:15:46.212929 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.212561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35484)
0924 07:15:46.212745 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.212827 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:15:46.213672 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.213452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35492)
0924 07:15:46.213553 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.213606 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:15:46.213814 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
I20260924 07:15:46.224370 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 1
I20260924 07:15:46.225189 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.224904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35494)
0924 07:15:46.225050 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.225109 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 07:15:46.225952 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.225741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35504)
0924 07:15:46.225848 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.225893 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 07:15:46.226073 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
W20260924 07:15:46.280512 22805 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:46.280838 22805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:46.280864 22805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:46.280928 22805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:46.285156 22805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:46.285224 22805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:46.285243 22805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:46.285259 22805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:46.285274 22805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:46.288990 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 2
I20260924 07:15:46.289830 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.289513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35508)
0924 07:15:46.289668 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.289736 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 07:15:46.290573 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.290371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35510)
0924 07:15:46.290466 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.290514 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:15:46.290701 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
I20260924 07:15:46.291203 22805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44167,127.0.98.61:42079,127.0.98.60:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:46.292315 22805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:46.293354 22805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:15:46.294239 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 2
I20260924 07:15:46.294862 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.294627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35522)
0924 07:15:46.294744 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.294797 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:15:46.295626 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.295349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35532)
0924 07:15:46.295501 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.295556 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 07:15:46.295774 22799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43893: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43893: connect: Connection refused (error 111)
W20260924 07:15:46.301914 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:46.301915 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:46.301914 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:46.302291 22805 server_base.cc:1034] running on GCE node
I20260924 07:15:46.302913 22805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:46.304126 22805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:46.305298 22805 hybrid_clock.cc:648] HybridClock initialized: now 1790234146305270 us; error 45 us; skew 500 ppm
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:46.308038 22805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:15:46.308964 22805 webserver.cc:460] Webserver started at http://127.0.98.60:40631/ using document root <none> and password file <none>
I20260924 07:15:46.309495 22805 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:46.309551 22805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:46.309712 22805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:46.311748 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data/instance:
uuid: "628bff9618fe487694cc30c6dac8b518"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "aaf1f7ea4e15fcf9066de8edcb1366f5"
server_key_iv: "715f1b7fcd48111c6420a0d46ec3502b"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.312274 22805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal/instance:
uuid: "628bff9618fe487694cc30c6dac8b518"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "aaf1f7ea4e15fcf9066de8edcb1366f5"
server_key_iv: "715f1b7fcd48111c6420a0d46ec3502b"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.316411 22805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:15:46.318928 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:46.320302 22805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:15:46.320412 22805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "628bff9618fe487694cc30c6dac8b518"
format_stamp: "Formatted at 2026-09-24 07:15:46 on dist-test-slave-cnv3"
server_key: "aaf1f7ea4e15fcf9066de8edcb1366f5"
server_key_iv: "715f1b7fcd48111c6420a0d46ec3502b"
server_key_version: "encryptionkey@0"
I20260924 07:15:46.320533 22805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:46.334239 22805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:46.337414 22805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:46.337591 22805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:46.344497 22805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43893
I20260924 07:15:46.344830 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43893 every 8 connection(s)
I20260924 07:15:46.345484 22805 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:15:46.348663 22875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:46.352516 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:46.353442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 22805
I20260924 07:15:46.353541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233866113376-392-0/minicluster-data/master-2/wal/instance
I20260924 07:15:46.353793 392 external_mini_cluster.cc:1428] Setting key 80dbddc0643fd6d32c47c2c7e1394cdf
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:46.366056 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.354439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.0.98.62:44167)
0924 07:15:46.354567 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.354571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.355644 (+ 1073us) server_negotiation.cc:407] Connection header received
0924 07:15:46.356617 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.356621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.356681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.356749 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.357594 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.358112 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.359084 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.359293 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.359403 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.362049 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.362052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.362054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.362081 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.363660 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.364268 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.364271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.364273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.364375 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.364700 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.364703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.364705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.364947 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.365046 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.365711 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.365884 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:15:46.366981 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.354904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.0.1:39900)
0924 07:15:46.355503 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.355546 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.356360 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.356820 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.356825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.357195 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.357443 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.357456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.358208 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.358212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.358970 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.358978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.359452 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.359456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.359628 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.360177 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.360197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.361860 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.363768 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.363775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.363787 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.364139 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.364469 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.364472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.364473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.364611 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.365049 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.365058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.365397 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.365569 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.365889 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":127,"threads_started":1}
I20260924 07:15:46.366981 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.354902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.0.98.60:43893)
0924 07:15:46.355501 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.355519 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.355555 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:15:46.356327 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.356345 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.356537 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.356705 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.357505 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.358428 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.359234 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.359625 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.359722 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.362445 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.362451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.362462 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.362483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.364271 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.364732 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.364736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.364738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.364802 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.365160 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.365163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.365165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.365446 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.365640 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.365934 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.366052 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":164,"threads_started":1}
I20260924 07:15:46.368362 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:46.376694 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.368896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42079 (local address 127.0.0.1:33336)
0924 07:15:46.369095 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.369106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.369230 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.369554 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.369557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.369724 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.369828 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.369834 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.370587 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.370591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.371135 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.371142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.371631 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.371635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.371732 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.372194 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.372209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.373421 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.375174 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.375178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.375180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.375415 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.375671 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.375674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.375675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.375750 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.376177 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.376180 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.376277 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.376363 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.376506 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 07:15:46.376694 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.369078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.0.98.61:42079)
0924 07:15:46.369221 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.369226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.369240 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:15:46.369409 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.369412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.369464 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.369538 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.369938 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.370491 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.371249 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.371531 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.371620 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.373580 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.373585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.373587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.373618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.375047 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.375502 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.375506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.375508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.375587 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.375828 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.375831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.375833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.376011 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.376178 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.376422 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.376535 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:15:46.376922 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.369442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.0.98.62:44167)
0924 07:15:46.369562 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.369566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.369579 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:15:46.369640 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.369642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.369682 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.369744 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.370333 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.370828 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.371598 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.371819 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.371903 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.374043 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.374047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.374050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.374077 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.375417 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.375796 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.375800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.375801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.375879 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.376125 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.376129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.376131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.376320 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.376553 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.376669 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.376778 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:15:46.377758 22875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:46.385713 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.378427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35536)
0924 07:15:46.378582 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.378595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.378666 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.378838 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.378840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.378993 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.379074 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.379079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.379877 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.379879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.380374 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.380383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.380937 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.380940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.381011 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.381395 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.381407 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.382679 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.384302 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.384305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.384308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.384481 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.384715 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.384718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.384720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.384791 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.385226 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.385229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.385322 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.385426 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.385576 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 07:15:46.385847 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.378487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.0.98.60:43893)
0924 07:15:46.378584 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.378588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.378632 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:15:46.378726 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.378729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.378760 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.378820 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.379181 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.379773 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.380473 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.380741 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.380833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.382799 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.382803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.382806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.382828 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.384196 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.384561 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.384565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.384567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.384636 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.384888 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.384892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.384894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.385084 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.385188 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.385485 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.385663 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":37}
I20260924 07:15:46.386790 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.378148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33340 (local address 127.0.98.61:42079)
0924 07:15:46.378252 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.378256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.378272 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:46.378306 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.378309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.378352 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.378423 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.379019 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.379621 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.380408 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.380652 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.380733 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.383426 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.383431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.383434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.383460 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.384758 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.385138 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.385143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.385144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.385245 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.385485 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.385488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.385490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.385701 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.385920 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.386507 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.386640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:15:46.386987 22799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 3
I20260924 07:15:46.389837 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } attempt: 3
I20260924 07:15:46.390241 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:46.390610 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: Bootstrap starting.
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:46.391706 22697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1bf226c5f1e245daade783f8a0546266"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39910
I20260924 07:15:46.393565 22875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:46.394524 22875 log.cc:824] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:15:46 dist-test-slave-cnv3 krb5kdc[22643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:46.396067 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.387358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35546)
0924 07:15:46.387491 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.387505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.387615 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.387895 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.387898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.388087 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.388186 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.388191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.389028 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.389032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.389782 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.389792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.390335 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.390338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.390424 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.390915 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.390928 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.392422 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.394410 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.394413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.394416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.394657 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.394958 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.394962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.394964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.395055 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.395534 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.395538 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.395642 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.395743 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.395886 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 07:15:46.396132 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.387592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.0.98.60:43893)
0924 07:15:46.387680 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.387683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.387694 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:15:46.387724 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.387727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.387764 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.387871 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.388285 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.388915 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.389887 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.390207 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.390642 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.392561 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.392564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.392567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.392592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.394259 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.394765 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.394769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.394771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.394856 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.395144 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.395146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.395148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.395416 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.395505 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.395809 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.395926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:15:46.398149 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: No bootstrap required, opened a new log
I20260924 07:15:46.399428 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: Bootstrap starting.
I20260924 07:15:46.400110 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.390241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43893 (local address 127.0.0.1:35562)
0924 07:15:46.390431 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:46.390446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:46.390536 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:46.391270 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:46.391273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:46.391477 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:46.391586 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.391593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.392773 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.392777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.393547 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:46.393561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.394214 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.394218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:46.394310 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.394733 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:46.394747 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:46.396136 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:46.398326 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.398329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.398331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.398607 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.398929 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:46.398932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:46.398934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:46.399040 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:46.399554 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:46.399557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:46.399682 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 07:15:46.399793 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:15:46.399944 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 07:15:46.400388 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.390284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35562 (local address 127.0.98.60:43893)
0924 07:15:46.391027 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.391038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.391051 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:15:46.391082 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.391084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.391121 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.391690 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.391794 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.392648 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.393736 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.394092 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.394186 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.396328 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.396332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.396334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.396356 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.398045 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.398720 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.398723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.398725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.398825 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.399135 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.399138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.399139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.399357 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.399588 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.399858 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.399978 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":128,"threads_started":1}
I20260924 07:15:46.401922 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:46.402136 22875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.402398 22875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628bff9618fe487694cc30c6dac8b518, State: Initialized, Role: FOLLOWER
I20260924 07:15:46.402599 22799 log.cc:824] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:46.402720 22875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.403486 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: Bootstrap starting.
I20260924 07:15:46.404124 22875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:46.404536 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: No bootstrap required, opened a new log
I20260924 07:15:46.406224 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:46.407903 22724 log.cc:824] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:46.408721 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.408991 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be488fcd23084d089ff53267555d47f4, State: Initialized, Role: FOLLOWER
I20260924 07:15:46.409353 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.410825 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:15:46.410894 22894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:46.410966 22894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:46.408149 22805 master_runner.cc:186] Error getting master registration for 127.0.98.62:44167: OK, instance_id { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" instance_seqno: 1790234145925197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:15:46.411126 22805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:15:46.411724 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.411878 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.403216 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.413118 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.415153 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: No bootstrap required, opened a new log
W20260924 07:15:46.415985 22908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:46.416064 22908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:46.414535 22731 master_runner.cc:186] Error getting master registration for 127.0.98.62:44167: OK, instance_id { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" instance_seqno: 1790234145925197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:15:46.417733 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:15:46.418203 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.418453 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf226c5f1e245daade783f8a0546266, State: Initialized, Role: FOLLOWER
I20260924 07:15:46.418776 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.419306 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.419538 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.420173 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:46.420217 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:15:46.420310 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:46.421633 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
W20260924 07:15:46.421806 22677 tablet.cc:2369] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:15:46.422696 22697 leader_election.cc:288] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [CANDIDATE]: Term 1 election: Requested vote from peers be488fcd23084d089ff53267555d47f4 (127.0.98.61:42079), 628bff9618fe487694cc30c6dac8b518 (127.0.98.60:43893)
I20260924 07:15:46.423355 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bf226c5f1e245daade783f8a0546266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "628bff9618fe487694cc30c6dac8b518"
I20260924 07:15:46.423642 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:46.425041 22849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf226c5f1e245daade783f8a0546266 in term 1.
I20260924 07:15:46.425482 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1bf226c5f1e245daade783f8a0546266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be488fcd23084d089ff53267555d47f4"
I20260924 07:15:46.425719 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:46.427003 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bf226c5f1e245daade783f8a0546266 in term 1.
I20260924 07:15:46.427558 22673 leader_election.cc:302] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf226c5f1e245daade783f8a0546266, be488fcd23084d089ff53267555d47f4; no voters:
I20260924 07:15:46.428335 22910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:46.428586 22910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [term 1 LEADER]: Becoming Leader. State: Replica: 1bf226c5f1e245daade783f8a0546266, State: Running, Role: LEADER
I20260924 07:15:46.428946 22910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } }
I20260924 07:15:46.430802 22656 master_runner.cc:263] No leader master found from master 1bf226c5f1e245daade783f8a0546266
I20260924 07:15:46.430867 22656 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:15:46.431066 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf226c5f1e245daade783f8a0546266. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.431182 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:46.431557 22928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:46.434834 22928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:46.441740 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.432457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.0.98.60:43893)
0924 07:15:46.432577 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.432582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.433116 (+ 534us) server_negotiation.cc:407] Connection header received
0924 07:15:46.433315 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.433318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.433367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.433415 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.434408 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.435145 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.436385 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.436708 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.436784 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.438202 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.438205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.438209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.438237 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.439833 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.440309 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.440313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.440315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.440424 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.440719 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.440722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.440724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.440940 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.441056 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.441419 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.441544 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:15:46.444440 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.431754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.62:44167)
0924 07:15:46.431870 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.431885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.435637 (+ 3752us) server_negotiation.cc:407] Connection header received
0924 07:15:46.435800 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.435804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.435848 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.435892 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.436844 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.437603 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.438694 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.439025 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.439123 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.440709 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.440713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.440715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.440742 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.442270 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.443080 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.443084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.443086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.443184 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.443502 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.443505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.443507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.443714 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.443811 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.444153 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.444281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:15:46.449717 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:46.432207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.98.61:42079)
0924 07:15:46.432320 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.432324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.442264 (+ 9940us) server_negotiation.cc:407] Connection header received
0924 07:15:46.442453 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.442456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.442506 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.442552 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.443367 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.444140 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.445156 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:46.445446 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:46.445519 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:46.446954 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:46.446958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:46.446960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:46.446986 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:46.448269 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.448627 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.448631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.448633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.448707 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:46.448934 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:46.448937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:46.448939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:46.449114 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:46.449233 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:46.449451 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 07:15:46.449568 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:15:46.454836 22772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [term 1 FOLLOWER]: Refusing update from remote peer 1bf226c5f1e245daade783f8a0546266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:46.454871 22849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [term 1 FOLLOWER]: Refusing update from remote peer 1bf226c5f1e245daade783f8a0546266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:46.455760 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:46.456210 22910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:46.459615 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf226c5f1e245daade783f8a0546266. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.459635 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf226c5f1e245daade783f8a0546266. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.459753 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.459748 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.469719 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.469887 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:46.470199 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.470299 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266 [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:46.470337 22933 mvcc.cc:205] Tried to move back new op lower bound from 7332799063871545344 to 7332799063776403456. Current Snapshot: MvccSnapshot[applied={T|T < 7332799063871545344}]
I20260924 07:15:46.471647 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.471796 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.473430 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1bf226c5f1e245daade783f8a0546266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bf226c5f1e245daade783f8a0546266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } peers { permanent_uuid: "be488fcd23084d089ff53267555d47f4" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42079 } } peers { permanent_uuid: "628bff9618fe487694cc30c6dac8b518" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43893 } } }
I20260924 07:15:46.473570 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:46.474587 22934 mvcc.cc:205] Tried to move back new op lower bound from 7332799063871545344 to 7332799063776403456. Current Snapshot: MvccSnapshot[applied={T|T < 7332799063871545344}]
I20260924 07:15:46.474957 22935 mvcc.cc:205] Tried to move back new op lower bound from 7332799063871545344 to 7332799063776403456. Current Snapshot: MvccSnapshot[applied={T|T < 7332799063871545344}]
I20260924 07:15:46.480937 22928 catalog_manager.cc:1347] Generated new cluster ID: d3c6c36748e34e97acc6163ece3af0ac
I20260924 07:15:46.481016 22928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:46.502134 22928 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:46.504103 22928 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:46.524506 22928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bf226c5f1e245daade783f8a0546266: Generated new TSK 0
I20260924 07:15:46.525581 22928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:15:46.560746 22880 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.557646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.0.98.60:43893)
0924 07:15:46.557790 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.557795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.559599 (+ 1804us) server_negotiation.cc:407] Connection header received
0924 07:15:46.559801 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.559804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.559859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.559911 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.560600 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35574: BlockingRecv error: recv got EOF from 127.0.0.1:35574 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 07:15:46.562384 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.558248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.0.98.61:42079)
0924 07:15:46.558366 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.558370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.561260 (+ 2890us) server_negotiation.cc:407] Connection header received
0924 07:15:46.561440 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.561444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.561606 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.561681 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.562260 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33360: BlockingRecv error: recv got EOF from 127.0.0.1:33360 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:15:46.564308 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.558974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.62:44167)
0924 07:15:46.559091 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.559096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.563097 (+ 4001us) server_negotiation.cc:407] Connection header received
0924 07:15:46.563444 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.563450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.563514 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.563570 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.564177 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39932: BlockingRecv error: recv got EOF from 127.0.0.1:39932 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 07:15:46.569512 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.567723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.0.98.61:42079)
0924 07:15:46.567852 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.567856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.568378 (+ 522us) server_negotiation.cc:407] Connection header received
0924 07:15:46.568649 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.568653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.568742 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.568803 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.569374 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33374: BlockingRecv error: recv got EOF from 127.0.0.1:33374 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 07:15:46.575201 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.573347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.98.61:42079)
0924 07:15:46.573483 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.573487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.574080 (+ 593us) server_negotiation.cc:407] Connection header received
0924 07:15:46.574289 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.574293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.574348 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.574413 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.575079 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33390: BlockingRecv error: recv got EOF from 127.0.0.1:33390 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 07:15:46.581653 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.579892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.98.61:42079)
0924 07:15:46.580025 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.580030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.580609 (+ 579us) server_negotiation.cc:407] Connection header received
0924 07:15:46.580828 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.580831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.580888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.580944 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.581531 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33400: BlockingRecv error: recv got EOF from 127.0.0.1:33400 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 07:15:46.589846 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.588143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.61:42079)
0924 07:15:46.588291 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.588295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.588865 (+ 570us) server_negotiation.cc:407] Connection header received
0924 07:15:46.589069 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.589073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.589127 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.589176 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.589729 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33404: BlockingRecv error: recv got EOF from 127.0.0.1:33404 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 07:15:46.602041 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.600439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.61:42079)
0924 07:15:46.600560 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.600564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.601097 (+ 533us) server_negotiation.cc:407] Connection header received
0924 07:15:46.601285 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.601288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.601341 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.601401 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.601923 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33416: BlockingRecv error: recv got EOF from 127.0.0.1:33416 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 07:15:46.621680 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.620438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.61:42079)
0924 07:15:46.620634 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.620639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.620780 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:15:46.620967 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.620971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.621021 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.621091 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.621527 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33428: BlockingRecv error: recv got EOF from 127.0.0.1:33428 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 07:15:46.656961 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.655782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.61:42079)
0924 07:15:46.655909 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.655914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.656117 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:15:46.656265 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.656267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.656315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.656380 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.656844 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33438: BlockingRecv error: recv got EOF from 127.0.0.1:33438 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:15:46.724337 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.723068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.0.98.61:42079)
0924 07:15:46.723216 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.723221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.723423 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:15:46.723580 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.723582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.723632 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.723703 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.724195 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33448: BlockingRecv error: recv got EOF from 127.0.0.1:33448 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 07:15:46.855762 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:46.854575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.98.61:42079)
0924 07:15:46.854700 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:46.854704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:46.854865 (+ 161us) server_negotiation.cc:407] Connection header received
0924 07:15:46.854950 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:46.854953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:46.855021 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:46.855107 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:46.855654 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33460: BlockingRecv error: recv got EOF from 127.0.0.1:33460 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 07:15:47.115533 22806 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:47.114502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33468 (local address 127.0.98.61:42079)
0924 07:15:47.114650 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.114654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.114670 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:47.114781 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.114785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.114851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.114918 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:47.115368 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33468: BlockingRecv error: recv got EOF from 127.0.0.1:33468 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:15:47.412555 22894 catalog_manager.cc:1259] Loaded cluster ID: d3c6c36748e34e97acc6163ece3af0ac
I20260924 07:15:47.412690 22894 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: loading cluster ID for follower catalog manager: success
I20260924 07:15:47.414484 22894 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: acquiring CA information for follower catalog manager: success
I20260924 07:15:47.415736 22894 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 628bff9618fe487694cc30c6dac8b518: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:15:47.417236 22908 catalog_manager.cc:1259] Loaded cluster ID: d3c6c36748e34e97acc6163ece3af0ac
I20260924 07:15:47.417297 22908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: loading cluster ID for follower catalog manager: success
I20260924 07:15:47.418924 22908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: acquiring CA information for follower catalog manager: success
I20260924 07:15:47.420099 22908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be488fcd23084d089ff53267555d47f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:15:47.634783 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:47.630261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.98.61:42079)
0924 07:15:47.630576 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.630580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.630600 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:15:47.630674 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.630679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.630847 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.630962 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:47.631257 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.631884 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.632820 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.633062 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.633282 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:47.633683 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:47.634325 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:47.634441 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:47.634534 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:15:47.634600 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20260924 07:15:47.642678 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:47.638472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.0.98.60:43893)
0924 07:15:47.638729 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.638733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.638746 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:15:47.638849 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.638852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.639000 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.639121 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:47.639359 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.640037 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.640998 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.641249 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.641339 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:47.641598 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:47.642208 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:47.642311 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:47.642409 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:15:47.642481 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260924 07:15:47.650996 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:47.646517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33492 (local address 127.0.98.61:42079)
0924 07:15:47.647103 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.647109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.647333 (+ 224us) server_negotiation.cc:407] Connection header received
0924 07:15:47.647477 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.647481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.647549 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.647633 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:47.647929 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.648706 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.649813 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.650061 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.650146 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:47.650412 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:47.650506 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:47.650567 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:47.650740 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:15:47.650814 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442}
I20260924 07:15:47.651414 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:47.646553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.0.98.60:43893)
0924 07:15:47.646696 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.646701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.647075 (+ 374us) server_negotiation.cc:407] Connection header received
0924 07:15:47.647286 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.647290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.647349 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.647447 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:47.647730 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.648564 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.649488 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.649717 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.649814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:47.650874 (+ 1060us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:47.650964 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:47.651006 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:47.651171 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 07:15:47.651239 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:15:47.652186 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:47.646798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.0.98.62:44167)
0924 07:15:47.647058 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:47.647063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:47.647079 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:47.647200 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:47.647203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:47.647456 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:47.647590 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:47.647977 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.648560 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.650214 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:47.650465 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:47.650552 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:47.650942 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:47.651455 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:47.651551 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:47.651944 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 07:15:47.652027 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260924 07:15:47.660135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22656
I20260924 07:15:47.667488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22731
I20260924 07:15:47.675031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 22805
2026-09-24T07:15:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3011 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:15:47 dist-test-slave-cnv3 krb5kdc[23023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:15:47 dist-test-slave-cnv3 krb5kdc[23023](info): set up 2 sockets
Sep 24 07:15:47 dist-test-slave-cnv3 krb5kdc[23023](info): commencing operation
krb5kdc: starting...
W20260924 07:15:48.648212 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/test-user.keytab.
Sep 24 07:15:48 dist-test-slave-cnv3 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:15:48.774346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:15:48.886273 23036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:48.886602 23036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:48.886637 23036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:48.886703 23036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:48.890895 23036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:48.890974 23036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:48.891001 23036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:48.891023 23036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:48.891045 23036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:48.896700 23036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:48.897933 23036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:48.898947 23036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:48.907599 23041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:48.907601 23042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:48.907601 23044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:48.907656 23036 server_base.cc:1034] running on GCE node
I20260924 07:15:48.908391 23036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:48.909468 23036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:48.910647 23036 hybrid_clock.cc:648] HybridClock initialized: now 1790234148910621 us; error 41 us; skew 500 ppm
Sep 24 07:15:48 dist-test-slave-cnv3 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:48.913594 23036 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:15:48.914547 23036 webserver.cc:460] Webserver started at http://127.0.98.62:45583/ using document root <none> and password file <none>
I20260924 07:15:48.915087 23036 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:48.915146 23036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:48.915346 23036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:48.917773 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data/instance:
uuid: "988466dccb8a44a6b902dece0633bddd"
format_stamp: "Formatted at 2026-09-24 07:15:48 on dist-test-slave-cnv3"
server_key: "1cbddd238947cb4b42afa73f9653d801"
server_key_iv: "047daa10db9f1bb7cb7bc81ea410f108"
server_key_version: "encryptionkey@0"
I20260924 07:15:48.918334 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance:
uuid: "988466dccb8a44a6b902dece0633bddd"
format_stamp: "Formatted at 2026-09-24 07:15:48 on dist-test-slave-cnv3"
server_key: "1cbddd238947cb4b42afa73f9653d801"
server_key_iv: "047daa10db9f1bb7cb7bc81ea410f108"
server_key_version: "encryptionkey@0"
I20260924 07:15:48.922181 23036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 07:15:48.924739 23051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:48.925944 23036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:15:48.926065 23036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal
uuid: "988466dccb8a44a6b902dece0633bddd"
format_stamp: "Formatted at 2026-09-24 07:15:48 on dist-test-slave-cnv3"
server_key: "1cbddd238947cb4b42afa73f9653d801"
server_key_iv: "047daa10db9f1bb7cb7bc81ea410f108"
server_key_version: "encryptionkey@0"
I20260924 07:15:48.926168 23036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:48.951263 23036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:48.954535 23036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:48.954722 23036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:48.961628 23036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38367
I20260924 07:15:48.961921 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38367 every 8 connection(s)
I20260924 07:15:48.962615 23036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:15:48.965487 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:48.969503 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:48.971055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 23036
I20260924 07:15:48.971163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-0/wal/instance
I20260924 07:15:48.971474 392 external_mini_cluster.cc:1428] Setting key 3697f709a36de16168858d15bc79f22b
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 07:15:48 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:48.988056 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:48.972050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38367 (local address 127.0.0.1:34534)
0924 07:15:48.972783 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:48.972835 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:48.973694 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:48.974425 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:48.974431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:48.974885 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:48.975130 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:48.975143 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:48.976333 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:48.976335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:48.977088 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:48.977097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:48.977734 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:48.977738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:48.977845 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:48.978513 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:48.978530 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:48.982391 (+ 3861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:48.984687 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:48.984698 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:48.984711 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:48.985127 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:48.985514 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:48.985517 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:48.985520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:48.985692 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:48.986335 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:48.986343 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:48.986526 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 07:15:48.986733 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 07:15:48.987220 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":100,"spinlock_wait_cycles":54144,"thread_start_us":167,"threads_started":1}
I20260924 07:15:48.988055 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:48.972050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.0.98.62:38367)
0924 07:15:48.972569 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:48.972586 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:48.972974 (+ 388us) server_negotiation.cc:407] Connection header received
0924 07:15:48.974029 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:48.974034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:48.974246 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:48.974409 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:48.975275 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:48.976210 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:48.977200 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:48.977577 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:48.977729 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:48.982593 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:48.982601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:48.982617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:48.982648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:48.984546 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:48.985252 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:48.985257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:48.985258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:48.985414 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:48.985791 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:48.985794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:48.985795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:48.986216 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:48.986328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:48.986867 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 07:15:48.987133 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":353,"thread_start_us":153,"threads_started":1}
I20260924 07:15:48.990887 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:48.991745 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:48.991463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60456)
0924 07:15:48.991587 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:48.991669 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:15:48.992067 23054 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
I20260924 07:15:48.992986 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:48.992758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60460)
0924 07:15:48.992877 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:48.992924 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 07:15:48.993111 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:15:48.999791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39663
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:15:49.042387 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } attempt: 1
I20260924 07:15:49.043352 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.042979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60468)
0924 07:15:49.043160 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.043250 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 07:15:49.044195 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.043950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60476)
0924 07:15:49.044066 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.044121 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:15:49.044377 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
W20260924 07:15:49.111589 23111 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:49.111900 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:49.111923 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:49.111970 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 07:15:49.112569 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } attempt: 2
I20260924 07:15:49.113456 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.113127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60482)
0924 07:15:49.113307 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.113374 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 07:15:49.114133 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.113943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60486)
0924 07:15:49.114030 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.114076 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:15:49.114343 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
W20260924 07:15:49.116552 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:49.116613 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:49.116640 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:49.116671 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:49.116707 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:49.122349 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:49.123522 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:49.124529 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:49.133332 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:49.133332 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:49.133332 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:49.133759 23111 server_base.cc:1034] running on GCE node
I20260924 07:15:49.134341 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:49.135511 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:49.136673 23111 hybrid_clock.cc:648] HybridClock initialized: now 1790234149136679 us; error 62 us; skew 500 ppm
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:49.143901 23111 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:15:49.144984 23111 webserver.cc:460] Webserver started at http://127.0.98.61:38425/ using document root <none> and password file <none>
I20260924 07:15:49.145552 23111 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:49.145625 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:49.145833 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:49.148049 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data/instance:
uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "6fa287c7e5aa04bec83eb5315de03cc7"
server_key_iv: "ac008e7f4a8d56081309a835cdbe3393"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.148613 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal/instance:
uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "6fa287c7e5aa04bec83eb5315de03cc7"
server_key_iv: "ac008e7f4a8d56081309a835cdbe3393"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.152493 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:15:49.155125 23126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:49.156486 23111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:15:49.156625 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal
uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "6fa287c7e5aa04bec83eb5315de03cc7"
server_key_iv: "ac008e7f4a8d56081309a835cdbe3393"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.156749 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:49.198457 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:49.201797 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:49.201987 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:49.205513 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } attempt: 3
I20260924 07:15:49.206470 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.206058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60490)
0924 07:15:49.206300 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.206370 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260924 07:15:49.207304 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.207072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60506)
0924 07:15:49.207193 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.207246 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 07:15:49.207444 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39663: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39663: connect: Connection refused (error 111)
I20260924 07:15:49.209295 23111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39663
I20260924 07:15:49.209542 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39663 every 8 connection(s)
I20260924 07:15:49.210273 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:15:49.213189 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:49.216917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 23111
I20260924 07:15:49.217041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-1/wal/instance
I20260924 07:15:49.217336 392 external_mini_cluster.cc:1428] Setting key 4588adedcf802e94e2149f1b77ca16ed
I20260924 07:15:49.217175 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:15:49.233899 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.219538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.98.62:38367)
0924 07:15:49.219712 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.219716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.220898 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 07:15:49.221913 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.221917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.221979 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.222061 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.223245 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.224001 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.225391 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.225678 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.225772 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.229305 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.229309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.229311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.229339 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.230815 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.231485 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.231489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.231490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.231658 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.232095 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.232099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.232100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.232322 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.232435 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.233629 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.233766 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 07:15:49.234664 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.220108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38367 (local address 127.0.0.1:34542)
0924 07:15:49.220711 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.220763 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.221709 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.222236 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.222243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.222605 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.223097 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.223111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.224249 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.224253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.225232 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.225243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.225858 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.225861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.226123 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.226757 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.226774 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.229058 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.230945 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.230951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.230970 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.231337 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.231749 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.231752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.231754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.231946 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.232451 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.232459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.232851 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.233071 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.233482 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":154,"threads_started":1}
I20260924 07:15:49.236220 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:49.245906 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.236740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60520)
0924 07:15:49.236858 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.236872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.236955 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.237595 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.237598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.237776 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.237863 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.237869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.238888 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.238891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.239506 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.239515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.240114 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.240118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.240289 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.240800 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.240813 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.242238 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.244299 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.244302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.244305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.244502 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.244821 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.244824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.244826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.244903 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.245392 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.245395 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.245479 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.245572 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.245724 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":50}
I20260924 07:15:49.246071 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.236844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.98.61:39663)
0924 07:15:49.237218 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.237230 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.237252 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:15:49.237292 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.237296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.237452 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.237620 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.237962 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.238775 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.239606 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.240005 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.240101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.242376 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.242386 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.242401 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.242421 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.244179 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.244598 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.244604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.244606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.244683 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.244979 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.244982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.244984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.245298 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.245425 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.245681 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.245886 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":272,"thread_start_us":141,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:15:49.247327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasks9UDai/build/debug/bin/kudu
/tmp/dist-test-tasks9UDai/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38779
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:15:49.248060 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:49.248888 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.248584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:46984)
0924 07:15:49.248723 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.248810 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 07:15:49.249138 23128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
I20260924 07:15:49.250191 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.249976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47000)
0924 07:15:49.250088 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.250133 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 07:15:49.250349 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38779: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
I20260924 07:15:49.299597 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } attempt: 1
I20260924 07:15:49.300572 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.300261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47016)
0924 07:15:49.300422 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.300487 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:15:49.301354 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.301136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47026)
0924 07:15:49.301250 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.301296 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 07:15:49.301569 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38779: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
I20260924 07:15:49.350647 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } attempt: 4
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:49.361104 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.351239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60526)
0924 07:15:49.351475 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.351490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.351570 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.351895 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.351897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.352072 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.352173 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.352179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.353205 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.353208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.353798 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.353810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.354395 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.354398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.354473 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.355096 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.355110 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.356726 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.359293 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.359296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.359298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.359629 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.359974 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.359977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.359978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.360070 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.360618 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.360621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.360724 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.360827 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.360958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 07:15:49.361205 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.351316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.98.61:39663)
0924 07:15:49.351500 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.351505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.351669 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:15:49.351724 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.351727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.351782 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.351884 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.352278 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.353074 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.353921 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.354286 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.354398 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.356893 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.356897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.356900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.356930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.359152 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.359743 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.359747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.359749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.359844 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.360176 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.360179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.360181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.360420 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.360527 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.360911 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.361044 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 07:15:49.362267 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:49.363128 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.362852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47038)
0924 07:15:49.363006 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.363064 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 07:15:49.363976 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.363762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47040)
0924 07:15:49.363870 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.363919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:15:49.364110 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38779: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
I20260924 07:15:49.369777 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } attempt: 2
I20260924 07:15:49.370638 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.370296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47044)
0924 07:15:49.370429 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.370557 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:15:49.371465 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.371180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47050)
0924 07:15:49.371315 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.371366 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 07:15:49.371665 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38779: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
W20260924 07:15:49.374897 23185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:15:49.375188 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:15:49.375216 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:15:49.375279 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:15:49.379470 23185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:15:49.379534 23185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:15:49.379554 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:15:49.379570 23185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:15:49.379583 23185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:15:49.385118 23185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38367,127.0.98.61:39663,127.0.98.60:38779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:15:49.386209 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:15:49.387223 23185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:15:49.395753 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:49.395777 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:15:49.395753 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:15:49.396106 23185 server_base.cc:1034] running on GCE node
I20260924 07:15:49.396680 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:15:49.397818 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:15:49.399016 23185 hybrid_clock.cc:648] HybridClock initialized: now 1790234149399003 us; error 38 us; skew 500 ppm
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:15:49.401860 23185 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:15:49.402773 23185 webserver.cc:460] Webserver started at http://127.0.98.60:43777/ using document root <none> and password file <none>
I20260924 07:15:49.403335 23185 fs_manager.cc:359] Metadata directory not provided
I20260924 07:15:49.403431 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:15:49.403653 23185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:15:49.405719 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data/instance:
uuid: "f4e84874162f4bd4af0058b48c029dff"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "e6e2976498b072677c1738610b241bba"
server_key_iv: "3db27ce5b8e13504cadf773280efc3aa"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.406265 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal/instance:
uuid: "f4e84874162f4bd4af0058b48c029dff"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "e6e2976498b072677c1738610b241bba"
server_key_iv: "3db27ce5b8e13504cadf773280efc3aa"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.410683 23185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 07:15:49.413668 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:15:49.414949 23185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:15:49.415066 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data,/tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal
uuid: "f4e84874162f4bd4af0058b48c029dff"
format_stamp: "Formatted at 2026-09-24 07:15:49 on dist-test-slave-cnv3"
server_key: "e6e2976498b072677c1738610b241bba"
server_key_iv: "3db27ce5b8e13504cadf773280efc3aa"
server_key_version: "encryptionkey@0"
I20260924 07:15:49.415170 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:15:49.423280 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } attempt: 1
I20260924 07:15:49.424249 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.423908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47056)
0924 07:15:49.424068 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.424153 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 07:15:49.424947 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.424729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47062)
0924 07:15:49.424831 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.424879 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:15:49.425058 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38779: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38779: connect: Connection refused (error 111)
I20260924 07:15:49.426409 23185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:15:49.429685 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:15:49.429862 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:15:49.436646 23185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38779
I20260924 07:15:49.436900 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38779 every 8 connection(s)
I20260924 07:15:49.437675 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:15:49.440526 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:15:49.444160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasks9UDai/build/debug/bin/kudu as pid 23185
I20260924 07:15:49.444272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasks9UDai/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233866113376-392-0/minicluster-data/master-2/wal/instance
I20260924 07:15:49.444535 392 external_mini_cluster.cc:1428] Setting key ccc8bd4eb29a584d563d124b210e3190
I20260924 07:15:49.444617 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:49.456830 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.446575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.0.98.62:38367)
0924 07:15:49.446696 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.446701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.447084 (+ 383us) server_negotiation.cc:407] Connection header received
0924 07:15:49.447691 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.447695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.447752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.447823 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.448652 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.449153 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.450108 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.450324 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.450407 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.453112 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.453115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.453118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.453144 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.454606 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.455260 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.455263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.455264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.455345 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.455728 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.455731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.455733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.455987 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.456085 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.456555 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.456678 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:15:49.457814 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.446047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.0.98.60:38779)
0924 07:15:49.446470 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.446489 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.446528 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:15:49.447399 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.447424 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.447649 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.447812 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.448645 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.449491 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.450345 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.450731 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.450821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.453380 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.453386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.453396 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.453416 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.455162 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.455624 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.455629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.455630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.455711 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.455946 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.455950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.455951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.456373 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.456550 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.456851 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.456998 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":279,"spinlock_wait_cycles":9472,"thread_start_us":127,"threads_started":1}
I20260924 07:15:49.457819 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.446507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38367 (local address 127.0.0.1:34554)
0924 07:15:49.446919 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.446980 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.447437 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.447891 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.447897 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.448250 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.448512 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.448525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.449251 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.449255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.449998 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.450007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.450456 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.450459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.450680 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.451243 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.451263 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.452948 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.454726 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.454733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.454744 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.455143 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.455455 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.455459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.455461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.455635 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.456088 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.456094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.456341 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.456497 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.456845 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":35,"thread_start_us":129,"threads_started":1}
I20260924 07:15:49.458992 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:49.462847 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } attempt: 3
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:49.468220 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.459583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.61:39663)
0924 07:15:49.459753 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.459757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.459792 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:15:49.459915 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.459918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.459963 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.460030 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.460483 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.461164 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.462338 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.462671 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.462780 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.465087 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.465091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.465094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.465117 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.466516 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.466962 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.466966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.466967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.467041 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.467336 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.467339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.467341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.467559 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.467642 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.467951 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.468059 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 07:15:49.468778 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.459492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39663 (local address 127.0.0.1:60540)
0924 07:15:49.459738 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.459752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.459836 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.460049 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.460051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.460236 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.460361 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.460369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.461313 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.461316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.462176 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.462187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.462817 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.462821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.462927 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.463508 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.463525 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.464964 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.466622 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.466626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.466628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.466853 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.467155 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.467158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.467160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.467248 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.467642 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.467645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.467737 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.467873 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.468254 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 07:15:49.469512 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:15:49.471481 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.460419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.0.98.62:38367)
0924 07:15:49.461347 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.461351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.461361 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:15:49.461393 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.461395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.461448 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.461507 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.462386 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.463118 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.464155 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.464450 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.464534 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.467731 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.467735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.467738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.467764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.469629 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.470064 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.470068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.470069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.470380 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.470681 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.470685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.470687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.470899 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.471134 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.471210 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.471310 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853}
I20260924 07:15:49.472106 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.463569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47078)
0924 07:15:49.463664 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.463677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.463747 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.464008 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.464011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.464184 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.464273 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.464279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.465190 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.465194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.465707 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.465716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.466278 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.466282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.466368 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.466788 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.466800 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.468051 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.469699 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.469703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.469705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.469937 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.470217 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.470221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.470223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.470309 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.471642 (+ 1333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.471645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.471738 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.471825 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.471954 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:15:49.472134 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.463728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.0.98.60:38779)
0924 07:15:49.463832 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.463836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.463846 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:15:49.463876 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.463877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.463909 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.463975 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.464392 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.465092 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.465809 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.466153 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.466257 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.468186 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.468190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.468192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.468213 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.469558 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.470038 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.470041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.470043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.470120 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.470405 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.470407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.470408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.470580 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.470674 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.471904 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.472011 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:49.475652 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: Bootstrap starting.
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:15:49.478555 23179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:49.479187 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.470008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47092)
0924 07:15:49.470186 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.470199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.470256 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.470828 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.470830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.470996 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.471082 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.471089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.472580 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.472583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.473278 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.473288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.473798 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.473801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.473889 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.474323 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.474333 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.475696 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.477611 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.477614 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.477616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.477839 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.478135 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.478139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.478141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.478228 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.478699 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.478702 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.478776 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.478858 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.479003 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 07:15:49.479219 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.470064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.0.98.60:38779)
0924 07:15:49.470641 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.470650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.470661 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:15:49.470691 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.470694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.470733 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.470794 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.471188 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.472480 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.473392 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.473682 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.473775 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.475827 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.475831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.475834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.475855 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.477485 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.477941 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.477945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.477947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.478040 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.478356 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.478360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.478362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.478580 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.478668 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.478913 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.479063 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":511,"thread_start_us":110,"threads_started":1}
I20260924 07:15:49.479513 23179 log.cc:824] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:49.481456 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: No bootstrap required, opened a new log
I20260924 07:15:49.482220 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.473065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.0.98.61:39663)
0924 07:15:49.473161 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.473165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.473234 (+ 69us) server_negotiation.cc:407] Connection header received
0924 07:15:49.473358 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.473360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.473400 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.473464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.474089 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.474739 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.475830 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.476117 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.476221 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.478836 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.478840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.478843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.478868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.480413 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.480865 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.480869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.480870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.480959 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.481241 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.481245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.481246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.481467 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.481614 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.481968 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.482081 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:15:49.482569 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: Bootstrap starting.
I20260924 07:15:49.485110 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:49.485046 23179 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.485277 23179 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd36e4e6d8f24a47a4a4094b5e9d6afb, State: Initialized, Role: FOLLOWER
I20260924 07:15:49.485610 23179 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.486019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:15:49.486052 23254 log.cc:824] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:49.486155 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.486279 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.486927 23179 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:49.488121 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: No bootstrap required, opened a new log
I20260924 07:15:49.487286 23078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "988466dccb8a44a6b902dece0633bddd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34570
W20260924 07:15:49.491904 23273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:49.491973 23273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:49.491997 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.492209 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4e84874162f4bd4af0058b48c029dff, State: Initialized, Role: FOLLOWER
I20260924 07:15:49.492188 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } attempt: 2
I20260924 07:15:49.492524 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.490345 23111 master_runner.cc:186] Error getting master registration for 127.0.98.62:38367: OK, instance_id { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" instance_seqno: 1790234148959710 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:15:49.493508 23111 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:15:49.493786 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:49.496313 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.496579 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:15:49.499630 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:49.499752 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:49.497586 23185 master_runner.cc:186] Error getting master registration for 127.0.98.62:38367: OK, instance_id { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" instance_seqno: 1790234148959710 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:15:49.500840 23185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 07:15:49 dist-test-slave-cnv3 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:15:49.506412 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.492560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38779 (local address 127.0.0.1:47102)
0924 07:15:49.492796 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:15:49.492810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:15:49.492892 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:15:49.493473 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:15:49.493476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:15:49.493691 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:15:49.493799 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.493806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.494903 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.494906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.495797 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:15:49.495809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.496492 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.496496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:15:49.496589 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.501015 (+ 4426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:15:49.501032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:15:49.502663 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:15:49.504705 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.504709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.504711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.504988 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.505294 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:15:49.505298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:15:49.505300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:15:49.505394 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:15:49.505838 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:15:49.505841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:15:49.505977 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:15:49.506085 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:15:49.506238 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260924 07:15:49.506769 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.493032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47102 (local address 127.0.98.60:38779)
0924 07:15:49.493139 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.493143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.493155 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:15:49.493193 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.493196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.493239 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.493311 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.493944 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.494772 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.495931 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.496274 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.496370 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.502843 (+ 6473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.502847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.502850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.502879 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.504585 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.505104 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.505107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.505109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.505196 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.505501 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.505504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.505506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.505736 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.505826 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.506184 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.506381 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:15:49.510377 23104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: Bootstrap starting.
I20260924 07:15:49.513237 23104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: Neither blocks nor log segments found. Creating new log.
I20260924 07:15:49.514158 23104 log.cc:824] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: Log is configured to *not* fsync() on all Append() calls
I20260924 07:15:49.516420 23104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: No bootstrap required, opened a new log
I20260924 07:15:49.519526 23104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.519767 23104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988466dccb8a44a6b902dece0633bddd, State: Initialized, Role: FOLLOWER
I20260924 07:15:49.520102 23104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.520689 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.520962 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.521540 23104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:15:49.521590 23078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:15:49.521694 23078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:49.522900 23078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.523979 23078 leader_election.cc:288] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [CANDIDATE]: Term 1 election: Requested vote from peers dd36e4e6d8f24a47a4a4094b5e9d6afb (127.0.98.61:39663), f4e84874162f4bd4af0058b48c029dff (127.0.98.60:38779)
I20260924 07:15:49.525368 23153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "988466dccb8a44a6b902dece0633bddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb"
I20260924 07:15:49.525624 23153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:49.526983 23153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 988466dccb8a44a6b902dece0633bddd in term 1.
I20260924 07:15:49.527473 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "988466dccb8a44a6b902dece0633bddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4e84874162f4bd4af0058b48c029dff"
I20260924 07:15:49.527746 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:15:49.528123 23054 leader_election.cc:302] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 988466dccb8a44a6b902dece0633bddd, dd36e4e6d8f24a47a4a4094b5e9d6afb; no voters:
I20260924 07:15:49.529069 23228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 988466dccb8a44a6b902dece0633bddd in term 1.
I20260924 07:15:49.530222 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:15:49.530443 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [term 1 LEADER]: Becoming Leader. State: Replica: 988466dccb8a44a6b902dece0633bddd, State: Running, Role: LEADER
I20260924 07:15:49.530788 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } }
I20260924 07:15:49.532320 23036 master_runner.cc:263] No leader master found from master 988466dccb8a44a6b902dece0633bddd
I20260924 07:15:49.532379 23036 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:15:49.532685 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988466dccb8a44a6b902dece0633bddd. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.532799 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: This master's current role is: LEADER
W20260924 07:15:49.536454 23300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:15:49.536580 23300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:15:49.536705 23307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:15:49.537554 23307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:15:49.548939 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.534142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.61:39663)
0924 07:15:49.534260 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.534265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.539786 (+ 5521us) server_negotiation.cc:407] Connection header received
0924 07:15:49.539978 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.539982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.540034 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.540081 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.541010 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.541799 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.542945 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.543286 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.543359 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.545023 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.545027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.545030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.545060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.546835 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.547574 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.547579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.547581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.547669 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.547964 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.547968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.547970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.548186 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.548298 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.548629 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.548755 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:49.549796 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.533286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.0.98.62:38367)
0924 07:15:49.533397 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.533402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.536727 (+ 3325us) server_negotiation.cc:407] Connection header received
0924 07:15:49.536911 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.536914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.536961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.537009 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.538078 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.538864 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.540329 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.540684 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.540754 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.545852 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.545856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.545859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.545890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.547979 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.548435 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.548439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.548441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.548548 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.548834 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.548837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.548839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.549052 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.549173 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.549521 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.549643 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:15:49.555759 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:49.534581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47104 (local address 127.0.98.60:38779)
0924 07:15:49.534688 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.534692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.540258 (+ 5566us) server_negotiation.cc:407] Connection header received
0924 07:15:49.541616 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.541619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.541679 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.541725 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.543915 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.544687 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.546230 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:49.546596 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:49.546681 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:49.552334 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:15:49.552338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:15:49.552341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:15:49.552372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:15:49.553946 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.554434 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.554438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.554439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.554523 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:15:49.554794 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:15:49.554797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:15:49.554799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:15:49.555026 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:15:49.555143 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:49.555490 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 07:15:49.555610 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:15:49.557708 23228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [term 1 FOLLOWER]: Refusing update from remote peer 988466dccb8a44a6b902dece0633bddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:49.558367 23305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:49.559012 23153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [term 1 FOLLOWER]: Refusing update from remote peer 988466dccb8a44a6b902dece0633bddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:15:49.559604 23305 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:15:49.564112 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988466dccb8a44a6b902dece0633bddd. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.564236 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.565037 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 988466dccb8a44a6b902dece0633bddd. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.565163 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.568925 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.569082 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:49.569171 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.569228 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd [sys.catalog]: This master's current role is: LEADER
I20260924 07:15:49.569712 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.569829 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.570525 23312 mvcc.cc:205] Tried to move back new op lower bound from 7332799076580012032 to 7332799076482187264. Current Snapshot: MvccSnapshot[applied={T|T < 7332799076580012032}]
I20260924 07:15:49.570600 23313 mvcc.cc:205] Tried to move back new op lower bound from 7332799076580012032 to 7332799076482187264. Current Snapshot: MvccSnapshot[applied={T|T < 7332799076580012032}]
I20260924 07:15:49.570883 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "988466dccb8a44a6b902dece0633bddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988466dccb8a44a6b902dece0633bddd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38367 } } peers { permanent_uuid: "dd36e4e6d8f24a47a4a4094b5e9d6afb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39663 } } peers { permanent_uuid: "f4e84874162f4bd4af0058b48c029dff" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38779 } } }
I20260924 07:15:49.570984 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:15:49.571415 23314 mvcc.cc:205] Tried to move back new op lower bound from 7332799076580012032 to 7332799076482187264. Current Snapshot: MvccSnapshot[applied={T|T < 7332799076580012032}]
I20260924 07:15:49.577209 23307 catalog_manager.cc:1347] Generated new cluster ID: d235e8c42de6436fb5434135561e09dd
I20260924 07:15:49.577307 23307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:15:49.626606 23307 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:15:49.627974 23307 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:15:49.639470 23307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 988466dccb8a44a6b902dece0633bddd: Generated new TSK 0
I20260924 07:15:49.640465 23307 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:15:49.677003 23106 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.673735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.62:38367)
0924 07:15:49.673871 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.673876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.675886 (+ 2010us) server_negotiation.cc:407] Connection header received
0924 07:15:49.676085 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.676088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.676142 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.676194 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.676857 (+ 663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34582: BlockingRecv error: recv got EOF from 127.0.0.1:34582 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 07:15:49.678970 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.674537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.0.98.61:39663)
0924 07:15:49.674647 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.674651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.677530 (+ 2879us) server_negotiation.cc:407] Connection header received
0924 07:15:49.677952 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.677955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.678006 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.678056 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.678833 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60574: BlockingRecv error: recv got EOF from 127.0.0.1:60574 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 07:15:49.679752 23259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.675237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.0.98.60:38779)
0924 07:15:49.675400 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.675405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.677699 (+ 2294us) server_negotiation.cc:407] Connection header received
0924 07:15:49.678184 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.678187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.678235 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.678281 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.679621 (+ 1340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47112: BlockingRecv error: recv got EOF from 127.0.0.1:47112 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 07:15:49.685993 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.684069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.0.98.61:39663)
0924 07:15:49.684205 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.684211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.684867 (+ 656us) server_negotiation.cc:407] Connection header received
0924 07:15:49.685110 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.685114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.685208 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.685270 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.685877 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60590: BlockingRecv error: recv got EOF from 127.0.0.1:60590 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 07:15:49.691077 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.689588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.61:39663)
0924 07:15:49.689693 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.689697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.690181 (+ 484us) server_negotiation.cc:407] Connection header received
0924 07:15:49.690367 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.690370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.690421 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.690469 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.690958 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60592: BlockingRecv error: recv got EOF from 127.0.0.1:60592 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:15:49.696965 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.695396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.98.61:39663)
0924 07:15:49.695505 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.695510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.696058 (+ 548us) server_negotiation.cc:407] Connection header received
0924 07:15:49.696246 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.696249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.696300 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.696359 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.696850 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60594: BlockingRecv error: recv got EOF from 127.0.0.1:60594 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 07:15:49.704617 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.703239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.61:39663)
0924 07:15:49.703414 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.703418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.703618 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:15:49.703832 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.703836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.703892 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.703944 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.704501 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60604: BlockingRecv error: recv got EOF from 127.0.0.1:60604 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 07:15:49.715857 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.714645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.0.98.61:39663)
0924 07:15:49.714769 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.714774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.715009 (+ 235us) server_negotiation.cc:407] Connection header received
0924 07:15:49.715141 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.715143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.715205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.715271 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.715734 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60606: BlockingRecv error: recv got EOF from 127.0.0.1:60606 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 07:15:49.734918 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.733870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.0.98.61:39663)
0924 07:15:49.734065 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.734070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.734089 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:15:49.734253 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.734257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.734304 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.734377 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.734800 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60618: BlockingRecv error: recv got EOF from 127.0.0.1:60618 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260924 07:15:49.770085 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.768997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.0.98.61:39663)
0924 07:15:49.769155 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.769159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.769232 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:15:49.769419 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.769422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.769486 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.769569 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.769963 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60626: BlockingRecv error: recv got EOF from 127.0.0.1:60626 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260924 07:15:49.837275 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.836098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.0.98.61:39663)
0924 07:15:49.836273 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.836277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.836378 (+ 101us) server_negotiation.cc:407] Connection header received
0924 07:15:49.836549 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.836551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.836608 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.836675 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.837153 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60642: BlockingRecv error: recv got EOF from 127.0.0.1:60642 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 07:15:49.968509 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:49.967353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.0.98.61:39663)
0924 07:15:49.967494 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:49.967498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:49.967674 (+ 176us) server_negotiation.cc:407] Connection header received
0924 07:15:49.967825 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:49.967829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:49.967879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:49.967948 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:49.968398 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60656: BlockingRecv error: recv got EOF from 127.0.0.1:60656 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 07:15:50.227783 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:15:50.226734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.0.98.61:39663)
0924 07:15:50.226896 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.226900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.226916 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:50.227032 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.227035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.227081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.227156 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:15:50.227663 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60672: BlockingRecv error: recv got EOF from 127.0.0.1:60672 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 07:15:50.493305 23273 catalog_manager.cc:1259] Loaded cluster ID: d235e8c42de6436fb5434135561e09dd
I20260924 07:15:50.493413 23273 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: loading cluster ID for follower catalog manager: success
I20260924 07:15:50.495309 23273 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: acquiring CA information for follower catalog manager: success
I20260924 07:15:50.496588 23273 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd36e4e6d8f24a47a4a4094b5e9d6afb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:15:50.501047 23286 catalog_manager.cc:1259] Loaded cluster ID: d235e8c42de6436fb5434135561e09dd
I20260924 07:15:50.501122 23286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: loading cluster ID for follower catalog manager: success
I20260924 07:15:50.502761 23286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: acquiring CA information for follower catalog manager: success
I20260924 07:15:50.503968 23286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4e84874162f4bd4af0058b48c029dff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:15:50.745679 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:50.741788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.0.98.61:39663)
0924 07:15:50.742060 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.742064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.742080 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:15:50.742201 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.742204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.742324 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.742429 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:50.742666 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.743235 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.744198 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.744435 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.744527 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:50.744859 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:50.745322 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:50.745413 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:50.745489 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 07:15:50.745545 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
I20260924 07:15:50.752892 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:50.748976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.0.98.60:38779)
0924 07:15:50.749242 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.749247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.749304 (+ 57us) server_negotiation.cc:407] Connection header received
0924 07:15:50.749526 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.749530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.749652 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.749756 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:50.750013 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.750540 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.751412 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.751638 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.751726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:50.752044 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:50.752489 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:50.752598 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:50.752676 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:15:50.752736 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260924 07:15:50.759330 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:50.755944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.0.98.60:38779)
0924 07:15:50.756117 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.756121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.756413 (+ 292us) server_negotiation.cc:407] Connection header received
0924 07:15:50.756614 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.756618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.756667 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.756740 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:50.756964 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.757462 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.758388 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.758606 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.758689 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:50.758918 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:50.758988 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:50.759044 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:50.759139 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:15:50.759192 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 07:15:50.760021 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:50.756084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60684 (local address 127.0.98.61:39663)
0924 07:15:50.756203 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.756207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.756314 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:15:50.756452 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.756456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.756516 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.756579 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:50.756857 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.757466 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.758730 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.759074 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.759149 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:50.759540 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:50.759619 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:50.759659 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:50.759821 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:15:50.759878 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:15:50.761397 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:15:50.755963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.62:38367)
0924 07:15:50.756555 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0924 07:15:50.756560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:15:50.756574 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:15:50.756661 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:15:50.756663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:15:50.756823 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:15:50.756960 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:15:50.757801 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.758603 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.759774 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:15:50.760265 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:15:50.760364 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:15:50.760657 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:15:50.761017 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:15:50.761103 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:15:50.761180 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 07:15:50.761238 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":74,"threads_started":1}
I20260924 07:15:50.764566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 23036
I20260924 07:15:50.770242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 23111
I20260924 07:15:50.775645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasks9UDai/build/debug/bin/kudu with pid 23185
2026-09-24T07:15:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3098 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6110 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284653 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST